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Overview 



The Talent Development Middle School model is a whole-school reform approach de- 
signed to improve student achievement in urban middle schools that serve high-poverty popula- 
tions. The model includes a systematic reorganization of schools into small learning communi- 
ties, in which teachers are part of interdisciplinary teams that share the same students and have 
common planning time. The model also offers an academic curriculum based on nationally rec- 
ognized standards, professional development opportunities for teachers, the use of curriculum 
coaches to help support teachers on an ongoing basis, and extra help for students struggling in 
mathematics or reading. 

MDRC is evaluating the model at the invitation of the organization that created it, the 
Center for Research on the Education of Students Placed At Risk (CRESPAR), based at The 
Johns Hopkins University. Funding for this report was provided, through CRESPAR, by the 
U.S. Department of Education’s Institute of Education Sciences. 

Covering the first three years of Talent Development’s operation in six middle schools 
as well as up to two additional years of follow-up for a subset of the schools, this report focuses 
on student achievement and attendance outcomes for seventh- and eighth-graders. It also de- 
scribes the context in which Talent Development operates, and it reviews the model's compo- 
nents and implementation. In general, the findings shed light on the effectiveness of an early 
phase of Talent Development’s expansion. 



Key Impact Findings 

• Talent Development had a positive impact on math achievement for eighth-grade students, 
which emerged in the third year of implementation and then strengthened during the next 
two years in the schools for which data are available. 

• Talent Development schools exhibited modest impacts on eighth-grade attendance rates. 

• The model produced an inconsistent pattern of impacts on eighth-grade reading achieve- 
ment: Modest improvements occurred in some years but not in others. 

• Talent Development did not produce a consistent pattern of impacts, positive or negative, 
on seventh-grade math or reading achievement or attendance. 

Taking the strengths and limitations of the model into account, and given the small num- 
ber of schools in the analysis and the limited follow-up period for most of the schools, it is impor- 
tant to be cautious about drawing definitive conclusions about the effectiveness of the model for 
this interim report. 

A subsequent report will examine, among other questions, whether impacts on mathe- 
matics achievement can be sustained after the initial implementation phase and whether greater 
effects on reading achievement and attendance emerge as the model matures. The report will 
also include findings for five additional later-implementing middle schools whose progress can 
provide an indication of the model’ s capacity to scale up effectively in a large urban district. 
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Preface 



Middle schools have been called the “weak link” in American education, and, for stu- 
dents in urban, low-income communities, this crucial phase between elementary school and 
high school is especially tenuous. Such students face significant obstacles to acquiring the foun- 
dation of knowledge and the skills that are required for success in high school and beyond. Yet 
even though the challenges that these students face are well documented, there is little rigorous 
research on interventions designed to improve the outcomes for middle school students. 

The Talent Development Middle School model was created to make a difference in 
struggling urban middle schools. The model is part of a trend in school improvement strategies 
whereby whole-school reform projects aim to improve performance and attendance outcomes 
for students through the use of major changes in both the organizational structure and the educa- 
tional processes of middle schools. The models that function in this way — broadly referred to 
as “comprehensive school reform (CSR) models” — have been developed both nationally and 
locally, and they receive support from a combination of federal, state, and local funding as well 
as from private foundations. 

Talent Development has been a key target of federal resources earmarked for expanding 
the use of CSR initiatives in middle schools. The model reflects many of the core principles 
embedded in the CSR movement. School-level structural changes, for example, create more 
personalized learning environments for students and teachers; curricular changes improve the 
rigor of coursework and raise teachers’ and students’ expectations; and professional develop- 
ment for teachers fills gaps in both content knowledge and pedagogy. 

The findings in this report — which offers an initial assessment of the first and most in- 
tensive effort at scaling up the use of the Talent Development Middle School model — indicate 
that Talent Development had a positive impact on eighth-grade math achievement and exhibited 
modest impacts on attendance rates. At the same time, the model produced an inconsistent pat- 
tern of impacts on eighth-grade reading and had few significant impacts on outcomes for sev- 
enth-grade students. This assessment is based on an innovative analytic methodology that relies 
on a combination of before-and-after and comparison-schools methods. 

Although the findings offer hope that the Talent Development model can improve aca- 
demic outcomes, at least in math, for middle school students, more data collection and analysis 
are needed before definite conclusions can be drawn. A subsequent report will track outcomes 
for two additional years of implementation and will provide a clearer picture of the potential for 
improvements in middle school achievement to lead to greater persistence in high school and, 
eventually, to graduation. 

Gordon Berlin 
President 
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Executive Summary 



Middle schools (grades 5 or 6 through grade 8) stand at a crucial intersection in Ameri- 
can public education systems. They are charged with the responsibility of building on the basic 
literacy and numeracy skills that students learn in elementary school and with helping students 
master those skills if they have fallen behind. They must be prepared to nurture the physical, 
social, and intellectual growth that students undergo in their early adolescent years. Moreover, 
they are expected to provide students with the habits of mind and behavior that they will need to 
make healthy transitions to high school and young adulthood. 

Yet too many middle schools are failing. In particular, schools that serve high-poverty 
student populations face significant obstacles that can keep them from providing adequate op- 
portunities for teaching and learning. Many students enter middle school, for example, with 
poor preparation in reading, writing, and mathematics, and the schools never succeed in narrow- 
ing the gap as students approach high school. The size and complexity of many middle schools 
make them ill-equipped to deal with discipline problems that interfere with effective teaching 
and learning. Patterns of poor attendance and weak study habits often begin for students during 
the middle grades and become established behavior patterns. Nonselective middle schools in the 
urban district that is the focus of this report exemplify these problems. 1 During the 1990s, for 
example, more than 75 percent of the seventh- and eighth-grade students in these schools had 
reading and math skills below grade level, and more than 25 percent could be considered 
chronic absentees (students with attendance rates of 80 percent or lower). 

Given these challenges, many students leave middle school to enter high school without the 
foundation of skills and work habits that they need for academic success. Throughout the 1990s, for 
example, almost all eighth-grade students in the district were promoted to the ninth grade on time. 
Yet only about 60 percent of those students were promoted to the tenth grade a year later, and less 
than 50 percent were on schedule to graduate four years after starting high school. 

The Talent Development Middle School model is a comprehensive reform initiative de- 
signed to help transform the structure and curriculum of large middle schools in urban districts, 
with the aim of improving students’ levels of achievement and raising teachers’ and students’ 
expectations. The model includes a systematic reorganization of each school into small learning 
communities, organized around interdisciplinary teacher teams that share the same students and 
have common planning time. It infuses the curriculum with academic courses in English, lan- 

1 In order to preserve the anonymity of the subjects in this study, the report refers to the participating school 
district as “the district.” The district includes 38 nonselective schools. The term “nonselective” refers to schools 
that typically enroll students from a nearby neighborhood and do not require them to meet academic or other 
performance standards for admission. 
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guage arts, mathematics, science, and U.S. history that are based on nationally recognized stan- 
dards. Teachers receive professional development on the use of the curriculum and accompany- 
ing instructional practice, and each school employs the services of curriculum coaches to help 
support teachers on an ongoing basis. The model provides catch-up opportunities during the 
school day to students who are struggling with mathematics or reading. 

Talent Development was created by practitioners and researchers at the Center for Re- 
search on the Education of Students Placed At Risk (CRESPAR), based at The Johns Hopkins 
University; the model operates in 21 middle schools nationwide. MDRC is conducting an inde- 
pendent, third-party evaluation of Talent Development, funded by the U.S. Department of Edu- 
cation’s Institute of Education Sciences, through CRESPAR. The evaluation and CRESPAR' s 
efforts to expand the use of Talent Development are part of the U.S. Department of Education’s 
Comprehensive School Reform Demonstration (CSRD) program. An earlier report by MDRC 
looked at the Talent Development High School model. 2 

This report focuses on preliminary findings for the impact of Talent Development on 
six nonselective, comprehensive middle schools that implemented the model in a northeastern, 
urban school district. (This district is the locus of Talent Development’s initial and most exten- 
sive scaling-up effort.) The six schools began working with Talent Development between 1996- 
1997 and 1998-1999 and are referred to throughout the report as “early-implementing schools.” 
As of the 2001-2002 school year, all six of the early-implementing schools had at least three 
years of implementation experience; four of these schools had four years of implementation 
experience; and two of those four schools had five years of implementation experience. Within 
each school, introduction of the core Talent Development components was usually a three- or 
four-year process. (The report also includes limited analysis for the first year of implementation 
in another five middle schools that began implementing the model in the district more recently.) 

In addition to assessing Talent Development’s impacts on seventh- and eighth-grade 
students’ achievement and attendance, the report describes the context in which the Talent De- 
velopment middle schools in the district operate, explains the model’s core components, and 
outlines the initial implementation of those components in the district. 



A Rigorous Approach to Assessing Impacts 

There are few rigorous studies of the effectiveness of comprehensive middle school re- 
form interventions. Because such reforms affect an entire school, the challenge in evaluating 
their impact lies in identifying a group of similar students in similar schools that did not experi- 



2 James J. Kemple and Corinne M. Herlihy, The Talent Development High School Model: Context, Components, 
and Initial Impacts on Ninth-Grade Students ’ Engagement and Performance (New York: MDRC, 2004). 



ES-2 




ence the intervention being tested and can provide a reliable “comparison” to tell evaluators 
what would have happened if business continued as usual. This type of comparison is important 
because many factors cause student outcomes (such as test scores, attendance rates, and grade 
promotion) to differ from school to school and from year to year. Rigorous comparison should 
determine how much variation was caused by a particular intervention — that is, what the im- 
pact of the intervention was over and above what would have occurred without the intervention. 
While a random assignment design — considered “the gold standard” of research evaluation — 
was not feasible for this study, the analytic approach used here includes a combination of two 
particularly strong quasi-experimental evaluation methods: an interrupted time series analysis 
and a comparison schools technique. The study relies on the strengths of each method to ad- 
dress the limitations that each one might have alone. 

In this report, two interrupted time series analyses are performed. The first compares stu- 
dent performance in Talent Development schools with the performance of similar students in the 
same schools prior to Talent Development’s implementation. The difference between perform- 
ance levels in the two groups is referred to as a “deviation from the baseline.” Many factors — 
some related to Talent Development and some not — may have contributed to the deviations from 
the baseline that emerged through this evaluation. In order to account for the factors that were un- 
related to Talent Development, a second interrupted time series analysis was conducted for non- 
Talent Development schools, a group of comparison schools in the same district that have charac- 
teristics similar to those of the Talent Development schools. The difference between the devia- 
tions from the baseline in the Talent Development schools and the deviations from the baseline in 
the non-Talent Development schools represents the estimated impact of Talent Development . 3 



Key Impact Findings 

The eighth grade marks the culmination of students’ middle school experiences and the 
start of a critical transition period, and eighth-graders' engagement and performance levels are key 
indicators of their readiness for the challenges of transitioning successfully to high school. For 
these reasons, the impact analysis in this report focuses first on outcomes for eighth-grade stu- 
dents. We also focus there because the model's estimated impact on the engagement and perform- 
ance of eighth-grade students represents Talent Development's cumulative effect on the middle 
school experience. In other words, in the first year of Talent Development's implementation, 



3 It should be noted, however, that even this combination of approaches may not control for all factors that 
may confound causal inferences about the effects of Talent Development on student performance. For exam- 
ple, the analytic approach may not account for systematic differences in school leadership's motivation to un- 
dertake a school change process and the influence that that may have had on both school functioning and stu- 
dent achievement, even without Talent Development’s components and supports. 
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eighth-grade students will have experienced only one year of the model; in the third year, some 
eighth-graders will have been exposed to the model in the sixth, seventh, and eighth grades. 

The findings for the study cover a five-year follow-up period. Because all six early- 
implementing schools had been using Talent Development for at least three of the follow-up 
years, the three-year follow-up results are the most reliable. The four-year follow-up results are 
based on the experiences of four of the early -implementing schools, and the five-year follow-up 
results are based on the experiences of just two of the schools. This means that, even though 
some of the impacts in Years 4 and 5 are promising, they may not be statistically significant 
because they are based on a smaller sample of schools. Years 4 and 5 of the follow-up period 
are important to the story of Talent Development, however. Implementation may need two or 
more years to gain enough traction to produce significant results, and eighth-graders in these 
later years may have benefited from cumulative years of exposure to Talent Development. 

• Talent Development had a positive impact on math achievement for 
eighth-grade students, which emerged in the third year of implementa- 
tion and then strengthened during the next two years in the schools for 
which data are available. 

Eighth-grade math achievement in both Talent Development and non-Talent Develop- 
ment schools improved during the follow-up period, relative to a common baseline period. Im- 
provements in the Talent Development schools began to outpace those in the comparison 
schools in Year 3. For example, during a three-year baseline period, eighth-grade students in the 
Talent Development schools scored at about the 23rd Normal Curve Equivalent (NCE) on the 
state standards assessment. 4 Students in the comparison schools scored at about the 24th NCE 
during the same period. In Year 3, average math scores improved to the 29th NCE in the Talent 
Development schools (a deviation of 6 NCE points), compared with the 28th NCE in the com- 
parison schools (a deviation of 4 NCE points). The difference of 2 points in deviations from the 
baseline is statistically significant and represents the impact of Talent Development. Among the 
schools for which data are available, this impact on math NCE scores grew to 4 points in both 
Years 4 and 5. 

More meaningful, perhaps, is Talent Development’s impact on reducing the percentage 
of eighth-grade students who scored in the bottom quartile for the state as a whole. In the pre- 
Talent Development period, about 83 percent of students in Talent Development schools and 81 
percent of students in non-Talent Development comparison schools fell into this category. By 

4 The Normal Curve Equivalent (NCE) is a way of measuring where a student falls along the normal curve 
for the statewide population of test-takers. The NCE score ranges from 1 to 99 and has a statewide average of 
50, which indicates that a student is performing at grade level. Less than 20 percent of students across the state 
have NCE scores below 30. 
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Year 3, this percentage dropped to 72 percent in the Talent Development schools (an 1 1 per- 
centage point deviation) and to 74 percent in the comparison schools (a 7 percentage point de- 
viation). The difference of 4 points in deviations from the baselines represents the impact of 
Talent Development. Although this impact is not statistically significant, it is part of a clear 
trend that grew over time to produce impacts of 12 percentage points in Year 4 and 1 1 percent- 
age points in Year 5 among the schools for which data are available. 5 

• Talent Development produced an inconsistent pattern of impacts on 
eighth-grade reading achievement over the follow-up period. 

Talent Development's impact on reading achievement was much less consistent than its 
impact on math achievement. The analysis found statistically significant and positive impact 
estimates for eighth-grade reading achievement in the second year of Talent Development im- 
plementation — an improvement of almost 3 NCEs on average reading achievement and a re- 
duction of nearly 6 percentage points in the percentage of eighth-graders scoring in the bottom 
quartile on the state reading assessment. However, these impacts dropped in Years 3 and 4 and 
then rose again in Year 5 for the two schools for which data are available. Only the impacts in 
Year 2 are statistically significant. 

• In general, Talent Development schools modestly outpaced their com- 
parison schools on eighth-grade attendance. 

During most years of follow-up, average eighth-grade attendance rates improved in 
both Talent Development and non-Talent Development comparison schools. Statistically sig- 
nificant impact estimates of about 2 percentage points are found in Years 2 and 3. Overall atten- 
dance rates averaged between 85 percent and 88 percent in both sets of schools during the five 
years of follow-up. 

• Talent Development did not produce a consistent pattern of impacts, 
positive or negative, on seventh-grade math achievement, reading 
achievement, or attendance rates during the five-year follow-up period. 

During most of the follow-up period, there was no systematic change, relative to the base- 
line period, in test scores or attendance rates among seventh-graders in the Talent Development 
schools. In some years, the Talent Development schools exhibited marginal improvements in 
math achievement, reading achievement, or attendance rates. In other years, there was no change, 
or even a slight decline in outcomes. These patterns were virtually the same for the non-Talent 

5 It is important to note that, for the first three years of Talent Development implementation, the pattern for 
the schools with data from Years 4 and 5 is the same as that for the other schools. This suggests that the find- 
ings in Years 4 and 5 are not simply driven by changes in the sample of schools included in the analysis that 
occurred after Year 3. 
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Development comparison schools. Overall, therefore, Talent Development did not produce im- 
pacts, positive or negative, on most of the test-score and attendance measures that were examined 
for seventh-graders. The one exception to this occurred in math problem-solving for seventh- 
graders. Here, during the last two years of the follow-up period, improvements in test scores for 
the Talent Development schools outpaced the more modest improvements in those scores for stu- 
dents in the non-Talent Development schools. 



Sizing Up the Results 

Taken together, what do these findings suggest about the Talent Development Middle 
School model? Two important patterns emerge: (1) The most prominent impacts occurred in 
mathematics achievement among eighth-grade students; and (2) the strength of impacts seems 
to be associated with the timing and intensity of Talent Development’s implementation. 

The most significant impacts were found for math achievement among eighth-grade 
students and were particularly strong in the later years of implementation. This pattern may re- 
flect a combination of factors: Eighth-graders in these later years may have benefited from cu- 
mulative years of exposure to Talent Development, and implementation may need two or more 
years to gain enough traction to produce significant results. Also, the development of math im- 
pacts before impacts in reading seems to be consistent with the nature of the curricular materials 
and teacher training in math, which focused on grade-specific content units and were readily 
transferable to classroom practice. 

The presence of Talent Development impacts seems to correspond with whether and 
when key components of the model were implemented. The components of the model were 
phased in over three years in the six schools that are the focus of this report. The impact analysis 
shows that improvements in student achievement, at least in math, began to emerge in Years 3 
and 4 of implementation. 

Although the early impact findings in this report should be considered preliminary — be- 
cause this study focuses on only six middle schools with three to five years of follow-up data — 
they are encouraging, particularly for math achievement among eighth-grade students. The magni- 
tude of the impacts reported here is, by traditional research standards, considered to be small to 
moderate. 6 However, the impacts are comparable to achievement impacts found in rigorous 
evaluations of other notable models of comprehensive school reform and to impacts found in the 

6 J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed (Hillsdale, NJ: Lawrence Erl- 
baum Associates, 1998); Howard S. Bloom, Sandra Ham, Laura Melton, and Julieanne O'Brien, Evaluating 
the Accelerated Schools Approach: A Look at Early Implementation and Impacts on Student Achievement in 
Eight Elementary Schools (New York: MDRC, 2001); Mark Lipsey, Design Sensitivity: Statistical Power for 
Experimental Research (Newbury Park, CA: Sage Publications, 1990). 
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Tennessee class-size experiment. 7 Finally, it is possible for small-to-moderate effect sizes to have 
substantial educational significance. For example, if all 38 nonselective middle schools in the dis- 
trict attained the most promising impacts on math achievement described in this report, more than 
1,200 eighth-grade students could move out of the bottom quartile in math achievement each year. 



Next Steps 

Overall, the findings in this report suggest that the Talent Development Middle School 
model has positive and significant impacts on certain measures, particularly when key compo- 
nents have been adequately implemented. This may offer promise that the model will have posi- 
tive and significant impacts on other outcomes in the future, but more data collection and analy- 
sis need to be done. 

A subsequent report from MDRC on the Talent Development Middle School model 
will track outcomes for two more years of implementation in the six early-implementing 
schools and for three years of implementation in the five later-implementing schools in an effort 
to answer remaining questions, including: 

• Will the improvements in eighth-grade mathematics be strengthened over 
time? Will the impacts continue to be sustained in early-implementing 
schools, and will those impacts eventually accrue in later-implementing 
schools and for students in other grade levels? 

• Will a more consistent pattern of impacts on eighth-grade reading achieve- 
ment, and on seventh-grade math and reading achievement, emerge in later 
follow-up years? 

• Will improvements in achievement during middle school years translate into 
students’ greater persistence in high school and their eventual graduation? 

The upcoming report, due in 2005, will be produced in the context of a range of com- 
prehensive school reform research sponsored by the U.S. Department of Education. Taken to- 
gether, this research has the potential to deliver a powerful message to policymakers, research- 
ers, and practitioners about what interventions help to improve student performance and atten- 
dance in low-performing secondary schools. 



7 Geoffrey D. Borman, Gina M. Hewes, Laura T. Overman, and Shelly Brown, “Comprehensive School 
Reform and Achievement: A Meta- Analysis,” Review of Educational Research 73 (2): 125-230; Jeremy D. 
Finn and Charles M. Achilles, “Tennessee’s Class Size Study: Findings, Implications, Misconceptions,” Edu- 
cational Evaluation and Policy Analysis 21 (2): 97-109. 



ES-7 




Introduction 

Large numbers of this nation’s secondary schools serving high-poverty student popula- 
tions are overwhelming failures. High proportions of students enter middle school (grades 5 or 6 
through grade 8) with poor preparation in reading, writing, and mathematics, and most do not 
narrow these gaps as they approach high school entry. Discipline problems with significant pro- 
portions of the early adolescent students in middle grades can create troubles that interfere with 
the general learning environment for all. Patterns of poor attendance and lack of attention to 
schoolwork and homework often begin for students during the middle grades and become estab- 
lished behavior patterns. In short, low-performing middle schools provide few opportunities for 
teaching and learning, and if students have not already reached the legal age for dropping out, 
they move to the next level without the foundation of skills and attitudes needed for success in 
high school. 

The Talent Development Middle School model is a comprehensive reform model for 
large middle schools that face serious problems with student attendance, discipline, and 
achievement. The model requires that schools reorganize into small learning communities and 
teaching teams. Schools adopt standards-based curricula in mathematics, language arts, science, 
and history, which are supported by ongoing, on-site professional development for teachers. 
Each of these changes is aimed specifically at enhancing student attendance in school, improv- 
ing student academic achievement, and preparing students for the transition to high school. The 
model began its first and most ambitious scaling-up effort in the large urban school district that 
is the focus of this report. 1 

The Talent Development Middle School model was designed by researchers, educators, 
and curriculum writers at the Center for Research on the Education of Students Placed At Risk 
(CRESPAR), based at The Johns Hopkins University, in collaboration with middle school prac- 
titioners. CRESPAR began work on the Talent Development Middle School model in 1995. 
The core features of the model were developed and pilot tested in a single middle school in the 
district beginning in the 1995-1996 school year. 2 The model has expanded to 21 middle schools 
in five states. 3 MDRC is conducting an independent third-party evaluation of the model, funded 
by the U.S. Department of Education, through its Institute of Education Sciences, as part of the 



'in order to preserve the anonymity of the subjects in this study, this report refers to the participating 
school district as “the district” and uses pseudonyms for individual schools. 

“Due to data consUaints and its status as a pilot school, this school was not included in the current analyses. 
3 Center for Social Organization of Schools/Talent Development Middle Schools Web site. 
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Comprehensive School Reform Demonstration program. An earlier report by MDRC looked at 
the Talent Development High School model and focused on the same district. 4 

This report provides a preliminary assessment of Talent Development’s impact on key 
outcomes for middle school students. It presents impact findings from Talent Development’s 
first five years of operation in 1 1 middle schools in a single school district (which serves more 
than 200,000 students). It focuses on student outcomes that are likely to be in greatest proximity 
to the early phases of Talent Development’s implementation: daily attendance, reading and 
math test scores (particularly for seventh- and eighth-grade students), and grade-level promotion 
through the first year of high school. 

• The report is organized into six sections: The first section provides evidence 
of the educational challenges faced by the middle schools in the large urban 
district where Talent Development began its work and where the middle 
school model has been used most pervasively. 

• The second section describes the Talent Development Middle School model, 
focusing on the features and components aimed at helping schools attack the 
problems they face as directly and immediately as possible. 

• The third section describes the implementation of the Talent Development 
Middle School model in 1 1 schools in the district that are the subject of the 
impact analysis. 

• The fourth section provides an overview of the data sources and analytic 
strategies being used to estimate the impacts that Talent Development has on 
a range of student outcomes. 

• The fifth section discusses the results that have emerged from analyses fo- 
cused on six early-implementing Talent Development middle schools, along 
with preliminary findings for the first year of implementation in five other 
Talent Development middle schools in the same district. 

• The final section discusses conclusions that may be drawn from the analyses, 
attempts to put these findings in the context of other comprehensive school 
reform models, and highlights upcoming research. 



4 Kemple and Herlihy, 2004. 
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The State of Middle Schools in the District 

The middle schools in the district are typical of the types of schools that Talent Devel- 
opment was specifically designed to help. Following is an overview of the characteristics of 
nonselective, 5 comprehensive middle schools in the district under study, just before Talent De- 
velopment began its work there. 

Large Populations of Minority and Low-Income Students 

Table 1 presents information for the group of 38 nonselective, comprehensive middle 
schools in the district. The table captures the state of these schools in the 1996-1997 school year 
— a year before Talent Development began to scale up the implementation of its model in sev- 
eral of the middle schools. At the time, these 38 nonselective middle schools enrolled nearly 
33,000 students in grades 6, 7, and 8, representing nearly 80 percent of the students in these 
grades throughout the district. 6 The table shows that about 71 percent of the students were 
black; 13 percent were white; 12 percent were of Hispanic origin; and the remaining students 
were of other racial/ethnic groups. About 8 percent of the students were classified for special 
education services. Over three-quarters were eligible for the federal free or reduced-price lunch 
program — an indication that these schools were predominantly serving students from low- 
income families. 

Low Student Attendance Rates 

Table 2 lists several indicators of school engagement and performance. These suggest 
that the schools faced high absentee rates, low levels of student achievement, and significant 
numbers of students failing to progress from one grade level to the next. The table shows that, 
among eighth-grade students, attendance rates averaged about 84 percent, indicating that the 
typical student was absent approximately 30 days during the 180-day school year. Over one- 
quarter of eighth-grade students might be classified as chronic absentees, having attendance 
rates of 80 percent or lower for the year. 



5 “Nonselective” schools typically enroll students from a nearby neighborhood and do not require them to 
meet academic or other performance standards for admission. 

6 Approximately 16 percent of the district’s sixth- through eighth-grade students attended schools covering 
grades 1 through 8, and about 3 percent attended “selective” middle schools that admit students on the basis of 
prior academic performance. About 1 percent of the students in these grades attended schools serving children 
with special needs. 
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The Talent Development Evaluation 

Table 1 



Characteristics of Nonselective Middle Schools in 
a Large, Urban School District, 

School Year 1996-1997 



Characteristic 


Nonselective 
Middle Schools 


Average number of students 


797 


Average number of students per grade 3 


6th grade 


250 


7 th grade 


269 


8th grade 


278 


Race/ethnicity (%) 


Black 


70.9 


White 


13.0 


Hispanic 


12.2 


Other 


3.8 


Gender (%) 


Male 


50.7 


Female 


49.3 


Classified for special education (%) 


8.4 


Eligible for free/reduced-price lunch (%) b 


76.5 


Total number of schools 


38 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 



NOTES: Sample includes students from 38 nonselective, comprehensive middle schools. Students in the 
sample were included on the district's test score records or were enrolled for at least 145 days during a given 
school year. 

“Several middle schools also serve students in the 5th grade. These students are not included in the table. 
b Calculated from the Common Core of Data for the 1999-2000 school year, provided by the U.S. 
Department of Education's National Center for Education Statistics. The figure shown represents the 
percentage of students who were eligible under the National School Lunch Act to participate in the federal 
free/reduced-price lunch program. 



- 4 - 








The Talent Development Evaluation 
Table 2 

Characteristics of Students in Nonselective Middle Schools in 
a Large, Urban School District, 

School Year 1996-1997 



Characteristic 


6th-Grade 

Students 


7th-Grade 

Students 


8th-Grade 

Students 


Overage for grade 0 (%) 


20.7 


19.8 


22.5 


Attendance rate b 


87.5 


85.6 


83.9 


Attendance rate of 90% or higher (%) 


56.4 


50.4 


46.8 


Attendance rate of 80% or lower (%) 


19.2 


25.2 


28.8 


SSA test scores 0 
Math 

Normal Curve Equivalent (NCE) score 


NA 


NA 


25.7 


Percentage scoring in the bottom quartile 


NA 


NA 


75.2 


Percentage scoring at or above grade level 


NA 


NA 


7.9 


Reading 


Normal Curve Equivalent (NCE) score 


NA 


NA 


29.3 


Percentage scoring in the bottom quartile 


NA 


NA 


66.3 


Percentage scoring at or above grade level 


NA 


NA 


12.4 


SAT-9 test scores d 
Math total 

Normal Curve Equivalent (NCE) score 


NA 


35.6 


35.7 


Percentage scoring in the bottom quartile 


NA 


56.8 


61.3 


Percentage scoring at or above grade level 


NA 


18.4 


17.2 


Reading comprehension 


Normal Curve Equivalent (NCE) score 


NA 


36.3 


37.0 


Percentage scoring in the bottom quartile 


NA 


51.0 


48.9 


Percentage scoring at or above grade level 


NA 


24.3 


25.0 


6th-grade promotion status 0 


7th grade in 1997-1998 school year (%) 


95.5 


NA 


NA 


8th grade in 1998-1999 school year (%) 


87.6 


NA 


NA 


Transfer status 

Present in same school in 1998-1999 school year (%) 


72.0 


NA 


NA 


7th-grade promotion status 0 


8th grade in 1997-1998 school year (%) 


NA 


96.1 


NA 


9th grade in 1998-1999 school year (%) 


NA 


89.6 


NA 


8th-grade promotion status 0 


9th grade in 1997-1998 school year (%) 


NA 


NA 


97.3 


10th grade in 1998-1999 school year (%) 


NA 


NA 


62.1 


12th grade in 2000-2001 school year (%) 


NA 


NA 


49.3 
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(continued) 







Table 2 (continued) 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes students from 38 nonselective, comprehensive middle schools. Students in the 
sample were included on the district's test score records or were enrolled for at least 145 days during a given 
school year. 

“Typically, students who were overage for grade were retained in the current grade or a prior one. “Overage 
for grade” means a student turned 12 before the start of the 6th grade, 13 before the start of the 7th grade, or 14 
before the start of the 8th grade. 

b Attendance rates were calculated for each student by dividing the number of days the student was present by 
the total number of days the student was enrolled in a given year. 

“State Standards Assessment (SSA) test scores were available only for the 8th grade. 

d SAT-9 test scores were available only for the 7th and 8th grades in the 1996-1997 school year. 

“Promotion status was calculated only for students who were listed on the district's administrative records in 
the 1997-1998 school year. Students were considered to be promoted if they were listed in the administrative 
file for the year indicated and were enrolled in the grade indicated. 



Poor Progress Through School 

Perhaps more distressing are the low rates at which students were progressing through 
school. In the 1996-1997 school year, about 12 percent of sixth- and seventh-grade students 
were not promoted to the next grade. Nearly 23 percent of the eighth-graders were overage for 
their grade, indicating that they had already been retained in their current grade or a previous 
one. Of the eighth-grade students who enrolled in a public high school in the district, 97 percent 
were promoted to ninth grade, but only 62 percent had been promoted to the tenth grade by the 
end of 1998-1999 school year. By the 2000-2001 school year — the year when this cohort of 
students was scheduled to graduate — less than 50 percent of these students were still enrolled 
in one of the district’ s high schools as twelfth-graders. 

Student Achievement Below Grade Level 

All eighth-grade students in the district are required to take the State Standards 
ment (SSA) test to determine their levels of proficiency in reading and math. The 
normed based on the performance of all students in the state. Table 2 indicates that the 
eighth-grade student in the district’s nonselective middle schools scored at the 26th 
Curve Equivalent (NCE) in math and at the 29th NCE in reading; the statewide average 



Assess- 
SSA is 
average 
Normal 
for both 
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tests is the 50th NCE. 7 Among the eighth-grade students, 8 percent scored at or above the state’s 
grade-level average in math, and only 12 percent scored at or above grade level in reading. A 
large majority of eighth-grade students scored in the bottom quartile in both math and reading. 

Table 2 also shows eighth-grade students’ performance on the Stanford Achievement 
Test, Version 9 (SAT-9) — a standardized test whose scores are based on the performance of 
students nationally. The table indicates that eighth-grade students in the district’s middle 
schools performed somewhat better when compared with other eighth-graders nationally than 
when compared with their peers in the state. The average scores for eighth-graders were about 
the 36th NCE in math and the 37th NCE in reading comprehension; the average nationally for 
both tests is the 50th NCE. Among eighth-grade students, 61 percent scored in the bottom quar- 
tile in math, and about 17 percent scored at or above the national grade-level average. In read- 
ing, about 49 percent of eighth-grade students scored in the bottom quartile, and 25 percent 
scored at or above the national grade-level average. In short, the data suggest that students in 
nonselective, comprehensive middle schools in the district are at high risk of leaving school 
with very low levels of literacy and numeracy. 

High Student Mobility 

Compounding the district’s low levels of student achievement and poor attendance are 
high rates of student mobility. The transfer status of sixth-graders provides an indication of stu- 
dent mobility: More than one-quarter of the district’s sixth-grade students in the 1996-1997 
school year were no longer enrolled in the same school two years later, and sixth-graders had 
transferred at least once to another public middle school in the district. This degree of student 
mobility amplifies the challenges that schools face in attempting to provide students with a sta- 
ble learning environment and consistent curricula and sets of instructional strategies. 

Finally, the averages presented in Tables 1 and 2 mask the variation among middle 
schools in the district, some of which serve somewhat more affluent communities and enable 
somewhat higher percentages of students to make adequate progress through school. As dis- 
cussed below, Talent Development aims specifically to serve students in the lowest-performing 
schools, many of which fall below the averages presented in Table 2. 8 



7 The Normal Curve Equivalent (NCE) is a way of measuring where a student falls along the normal curve. 
The normalized test score, which ranges from 1 to 99 with a mean of 50, allows for comparisons across tests 
and subjects. Unlike percentile rank scores, the NCE measurement has an equal interval between scores, which 
means that NCE scores can be averaged to allow for comparisons of groups of students or schools. 

8 Table 3 presents the mean characteristics of the district’s Talent Development middle schools. 
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How Talent Development Aims to Improve Middle Schools 

The researchers, educators, and curriculum writers who designed the Talent Develop- 
ment Middle School model conceived of a comprehensive paradigm for school reform that 
seeks to replace the common “talent-sorting” model of schooling, which classifies and tracks 
students based on demonstrated achievement. The Talent Development model asserts that all 
children can learn and must do so in an academic setting that is demanding and expresses high 
expectations. In the words of Wade Boykin, CRESPAR’s co-director, the model is aimed at 
“maximizing every child’s potential for academic development .” 9 

From the beginning, Talent Development’s central goals have been to help transform 
urban middle schools into strong learning institutions that provide every student with a stan- 
dards-based education and every teacher with the training, support, and materials needed to de- 
liver that. Ultimately, students are expected to achieve at “world class levels” and teachers to 
deliver “standards-based instruction in every lesson, every day”; specifically, after three years of 
implementation, the aim of the Talent Development Middle School model is “to have every 
eighth grader studying algebra, reading and analyzing great literature, performing hands-on sci- 
ence experiments, and interpreting original documents from our nation’s history .” 10 

To reach these goals, the Talent Development Middle School model addresses several 
key problems of high-poverty urban middle schools: the absence of strong curricula and the 
lack of well-prepared teachers; low expectations among students and their disengagement from 
school; poor academic performance, including students’ testing two or more years below grade 
level in reading and math; and the lack of supportive school-family and school-community rela- 
tionships. Overcoming these challenges in the middle years is important because up to half of 
students in large urban districts are unable to make a successful transition through the first years 
of high school . 11 

The Talent Development Middle School Model: Conceptual Framework 

As part of its early work with the CRESPAR team that is refining and scaling up the 
Talent Development model, MDRC constructed a conceptual framework describing Talent De- 
velopment’s theory of change. This research-based theory of change identifies the problems that 
Talent Development attempts to address, specifies the model’s core components, and defines 
the key short-term and long-term goals that it aspires to accomplish. Most important, the theory 
of change attempts to make explicit the pathways through which the core components of the 
model are intended to improve school functioning and, ultimately, student outcomes. 

9 Boykin, 2000, p. 7. 

l0 Center for Social Organization of Schools, 2002a. 

u See Neild and Weiss, 1999; Wilson and Corbett, 1999. 




Figure 1 presents a simplified version of this framework. At the top of the figure are 
elements of local contexts that are theorized to shape the ways that the Talent Development ap- 
proach has been adapted and sustained over time. These contextual elements include the poli- 
cies and administrative practices of states, school districts, and host schools, including funding 
and school finance structures. They also include the characteristics of the local labor market, 
employers, postsecondary education systems, students, families, teachers, and local communi- 
ties and the existing organizational structures and curricula of middle schools. Next, the figure 
highlights several of the most difficult problems that middle schools face. The left side of the 
figure then shows the four constructs that are affected by context and that outline the Talent De- 
velopment reform and its intended effects. These constructs lay out a process for reform that 
begins with the implementation of the model’ s structural elements and leads to changes in stu- 
dent performance outcomes. 

• Structural elements: changes in school organization, policies, curriculum 
content, resource allocations, and relationships with external entities that ad- 
dress key problems. The structural elements should be viewed as mecha- 
nisms that are mutually reinforcing and that offer direct and concrete ap- 
proaches to enhancing supports and learning opportunities. 

• Supports and learning opportunities: changes in school processes, teacher 
and student behaviors, experiences and expectations, and the use of internal 
and external resources that result from these structural elements. The sup- 
ports and learning opportunities are also mutually reinforcing and together 
are aimed at enhancing student performance through meditating outcomes. 

• Mediating outcomes: changes in students’ attitudes, engagement with 
school, and sense of efficacy and competence that emerge from changes in 
supports and learning opportunities. These mediating outcomes should be 
seen as direct antecedents to better performance outcomes. 

• Performance outcomes: changes in student achievement and progress to- 
ward promotion, high school graduation, and successful transitions to post- 
secondary education and employment. 

How the Model Addresses Key Problems: Components of Full 
Implementation 

Linkages among and between each stage in the framework illustrate the hypothesized 
pathways through which the Talent Development Middle School model is expected to affect 
student performance. In addition to representing the theory of change that drives the model, the 
framework can also be used to guide the measurement and analysis of the model’s impacts and 
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The Talent Development Evaluation 



Figure 1 

Simplified Conceptual Framework for the Talent Development Middle School Model 






















implementation. This section describes the components of full implementation, which are out- 
lined in Box 1; the following section discusses initial phases of implementation in the 11 
schools in this study. This conceptual framework is discussed below in terms of the problems 
that the Talent Development model aims to address. 



Box 1 

Key Components of Talent Development Implementation 

• Reorganization of the school into small learning communities and teaching 
teams 

• Implementation of standards-based, facilitated instructional programs in 

• Reading/English/Language Arts 

• Mathematics 

• Science 

• U.S. History 

• Provision of four tiers of continuous support for teachers, including 

• Subject-specific professional development with a focus on modeling 
lessons, content knowledge, instructional strategies, and classroom 
management 

• In-classroom support from a curriculum coach 

• In-school support from teachers who receive extra training 

• Support from CRESPAR-based instructional facilitators 

• Implementation of an elective replacement approach to provide extra help in 
mathematics and reading 

• Facilitation of school-family-community partnerships 



Addressing the Problem of Anonymity 

One reason often given by students for dropping out of school is the distance and es- 
trangement that they feel from teachers and administrators. 12 A positive school climate — in 
which students and adults know each other well and adults express care and concern for stu- 
dents’ welfare, intellectual growth, and educational success — is a key motivational element in 



12 See Altenbaugh, 1998. 
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the learning process for adolescents. 13 The large size of comprehensive secondary schools often 
depersonalizes the school environment, preventing teachers from working as teams and devel- 
oping an atmosphere that is conducive to learning. 14 The flux in classroom composition and stu- 
dent peer groups promotes anonymity and diminishes any sense of community. Students do not 
have a consistent group of teachers who are accountable for their success, and teachers do not 
have a chance to coordinate their coursework. The Talent Development model uses small learn- 
ing communities aimed at building a positive school climate that is characterized by personal- 
ized relationships among and between students and teachers. The Talent Development Middle 
School model’s small learning communities include: 

• School- within-a-school units of 200 to 300 students who occupy their own 
areas of the school and stay together for two or three years of middle school 

• Semi-departmentalization, whereby many teachers are responsible for two 
subjects, to limit the number of different students per teacher 

• Interdisciplinary teams of teachers who share the same students and have 
common planning time to address individual students’ needs 

• Mentor-advisors and looping, whereby some teachers or the entire team 
remain with the same students for multiple school years, to reinforce rela- 
tionships and a sense of responsibility for the success of each student 

Addressing the Problem of Low Student Expectations 

Adolescent students become bored, and attendance suffers, when they are not drawn to 
their coursework by the prospect of interesting and fulfilling class activities or when they see no 
connections between learning tasks and their own interests and goals. In traditional schools, the 
curriculum is usually separated into higher-level academic courses for college-bound students 
and lower-level academic and vocational courses for those presumed not to be college -bound. 
This separation often confines the teaching and learning process to the transfer of abstract 
knowledge from teachers to students or, in the case of vocational classes, to a narrow focus on 
specific job skills. There are usually very few opportunities to explore how basic skills are actu- 
ally applied outside the classroom. 15 

The Talent Development model involves organizational and instructional reforms that 
fight apathy by injecting meaning into the curriculum; connecting schoolwork to students’ 



13 See Wilson and Corbett, 1999. 

14 See Sizer, 1984; Hill, Foster, and Gendler, 1990; Powell, Cohen, and Farrar, 1985. 
15 See, for example. Resnick, 1987a; Raizen, 1989; Stasz et al., 1993; and Grubb, 1995. 
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backgrounds, interests, and goals; and enlivening lessons and learning activities with interesting 
and challenging applications. In Talent Development middle schools, this is achieved through: 

• Instruction in all academic areas featuring contextual learning, active 
student teams, and teaching for understanding 16 

• Reading, English, and Language Arts (RELA). All students work in 
cooperative teams. Student Team Literature — the chosen middle school 
language arts curriculum and instructional program — includes curricu- 
lar materials (partner discussion guides) to assist students’ study of high- 
quality fiction and nonfiction books. 

• Mathematics. A research- and standards-based mathematics curriculum 
is built around materials developed by the University of Chicago School 
Mathematics Project (UCSMP). The Talent Development Middle School 
Mathematics Program blends skill building with problem solving and is 
designed to enable all students to succeed in algebra in eighth grade. 

(Some Talent Development sites have incorporated standards-based 
mathematics curricula other than the UCSMP series.) 

• Science. The Talent Development middle school science staff help the 
science faculty at each school modify their curriculum so that it aligns 
with national science standards. Teachers implement hands-on modules 
drawn from a number of recently developed curricula. 

• U.S. History. The Talent Development middle school U.S. History cur- 
riculum brings together A History of US, a 10-volume series by Joy Ha- 
kim, and primary source materials in a series of student team-learning 
lessons. Firsthand accounts of participants are used to provide the social 
context of important events. 

• Coursework connected to students’ future career possibilities through a 
weekly class in which students explore career interests and understand the 
educational requirements for various kinds of occupations 

• Cultural relevance in curricular materials that include personalities and 
experiences from students’ own backgrounds 



16 For a description of the Talent Development middle school curriculum, see Center for Social Organiza- 
tion of Schools (2002b). 
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Addressing the Problem of Poor Prior Preparation 

One of the greatest challenges in U.S. education is how to produce student achievement 
at high performance standards during middle and high school grades in the face of the widely 
diverse prior preparations of the students who enter these levels. The Talent Development 
model does not compromise on standards; the model requires a common core curriculum for all 
students and heterogeneous grouping in these academic classes. It works to make a high- 
standards-based education a reality for all, by providing increased academic learning time and 
significant recovery opportunities for struggling students. 

• Elective-replacement approach to extra help. Talent Development middle 
schools have organized opportunities for extra help during the school day 
that are linked to the curriculum (replacing an elective or study hall for part 
of the year). Students who need extra help in math or reading attend a 10- 
week accelerated learning course (in addition to their regular math or reading 
courses) that uses cooperative groups and computers to provide intensive 
learning experiences. In a typical Talent Development middle school, this 
elective-replacement approach to extra help enables up to 300 students a year 
to receive an additional 10 to 12 weeks of instruction in mathematics and 
reading. 

• Professional development. Detracking (discontinuation of the practice of 
grouping students according to their ability levels) is further supported 
through professional development. Teachers receive training in subject- 
specific cooperative learning and classroom management techniques that are 
designed for diverse classrooms. 

Addressing the Problem of Limited Capacity to Implement Comprehensive 
Reform 

Currently, most schools — particularly those in highly stressed environments — have 
little or no capacity to address the problems of anonymity, low student expectations, and poor 
prior preparation of students. Even with specific strategies such as those listed above, imple- 
menting a comprehensive set of organizational reforms that respond to these challenges requires 
that teachers and administrators change their practice in fundamental ways. Hence, a critical 
component of the Talent Development approach is to provide schools with the following kinds 
of sustained and multilayered technical assistance and implementation support. 17 



17 For a more complete description of the institutional and professional development supports offered by 
Talent Development, see Center for Social Organization of Schools (2002c). 
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• A multiyear implementation plan. The Talent Development approach be- 
gins with a planning process of up to one year and culminates with faculty 
approval to move forward with the model. Sharing and planning between the 
school and Talent Development staff continue throughout the implementa- 
tion years. Typically, the model is phased in over three years, in addition to 
the planning year. Schools customize the model by incorporating existing 
strengths, and they localize it to support ongoing school, district, and com- 
munity initiatives. 

• Four layers of continuous support for teachers: 

• From 30 to 38 hours of professional development per subject per year 
for at least two years, with a focus on modeling upcoming lessons, con- 
tent knowledge, instructional strategies, and classroom management 

• Weekly in-classroom implementation support from a trained curricu- 
lum coach for each subject area (Typically, coaches are skilled and ex- 
perienced school district teachers or administrators who are placed on 
special assignment to the Talent Development model or are teachers or 
administrators from the school who are given release time.) 

• School-based support from lead teachers who receive additional training 

• Ongoing technical assistance from CRESPAR-based instructional fa- 
cilitators 

• Continuous support from the Talent Development team. Each school is 
assigned a support team of Talent Development trainers who maintain fre- 
quent contact with the school-based facilitators, principals, and key members 
of the school’s leadership and instructional teams. 

• Support from other schools in the Talent Development network. Each 
year, national conferences are held at which schools share their experiences 
and learn from one another. In addition, schools are linked to one another 
through Web-based electronic learning communities. 

Addressing the Problem of Schools’ Isolation from Families, Community, and 
Local Institutions 

Most schools are isolated from other institutions in their community, and many have 
very limited contact with students’ families (other than notifying them of severe disciplinary 
and academic problems). In addition, given the few connections between schools and commu- 
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nity institutions — particularly local employers — many students are inadequately informed 
about or prepared for the adult world and the world of work . 18 In short, schooling can become a 
process for isolating students from the world that they should be preparing to enter. Little effort 
is made to use the community as a resource for providing students with meaningful learning 
opportunities and as a context for highlighting the relevance of what they learn in school. 

To address such problems, through a partnership with the National Network of Partner- 
ship Schools (“the Network,” for short), Talent Development uses school-family -community 
partnerships . 19 The Network brings together schools, districts, and states that are committed to 
developing and maintaining comprehensive programs of school-family-community partner- 
ships. The goal is to enable families and communities to become informed about and involved 
in children’s education and in the schools. The Talent Development model draws on a variety of 
Network strategies for increasing these types of involvement, including: 

• Programs that provide parenting and child-rearing skills development 

• Communicating with families about school programs and student progress 

• Involving families as volunteers and audiences at the school 

• Involving families with their children in learning activities at home 

• Including families as participants in school decisions, governance, and so on 

• Collaborating to coordinate resources for families and students from the 
broader community as well as from families and students to the community 

The foregoing theory of change is a conceptual framework for describing the interacting 
components of the Talent Development Middle School model and illustrating how the model is 
expected to improve student outcomes. Though Talent Development is more prescriptive than 
other school reform approaches, CRESPAR strives to balance high-fidelity implementation of 
the model’s core components with the unique needs and circumstances of the participating 
schools and the capacities of school-based teachers and administrators. Even if the model is im- 
plemented intensively, a range of contextual factors at the school or district or state level can 
enhance or limit the model's capacity to make a positive difference for students. 



ls See, for example. Resnick, 1987b; Berryman and Bailey, 1992; Berryman, 1995. 
l9 See Center for Social Organization of Schools, 2002d. 
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The Context for Impacts: Implementation in Eleven Schools 

This section of the report describes the context for a preliminary assessment of Talent 
Development’s impacts on several outcomes illustrated in Figure 1. Whereas the prior section 
describes the model’s theory of change and the ideal components of full implementation, this 
section describes the actual implementation in 1 1 middle schools. (Box 1 summarizes the key 
components of Talent Development implementation.) This overview of the implementation of 
the model’s key components in the schools does not attempt to assess the quality or intensity of 
implementation because the evaluation does not include a systematic analysis of implementa- 
tion in these schools. 20 The goal of the overview is to shed light on hypotheses about when, 
where, and how one might expect Talent Development to begin making a difference for trou- 
bled middle schools and the students they serve. 

By the 1999-2000 school year, six middle schools in the district had adopted the Talent 
Development model. 21 These early-implementing schools are the primary focus of the impact 
analysis described below. Figure 2 presents a time line of implementation for the six schools. 
The figure shows that, as of the 2001-2002 school year (the most recent school year for which 
data are included in the analysis), 22 these early-implementing schools had been working with 
Talent Development for at least three years and for as many as five years (not including the ini- 
tial planning period). Five other middle schools in the district began implementing Talent De- 
velopment in the 2001-2002 school year. The analysis in this report captures only the first year 
of implementation for these schools, which are part of Talent Development’s most recent scal- 
ing-up effort in the district. 

The Planning Year and the Role of the Model Developer 

Talent Development middle schools usually begin by engaging with CRESPAR in a 
planning year to build faculty commitment to the model and to develop a concrete implementation 
timetable and strategy. However, several of the schools in this study had an abbreviated planning 
period that was limited to the spring semester, and one school, School B, did not have a planning 
period because the model did not gain sufficient faculty approval until shortly before implementa- 
tion began. CRESPAR' s goal is to help schools implement the model over two to three years, in- 
cluding all its major components (the four subject area programs, the recovery math and reading 
programs, training and supports to teachers and staff, and organizational modifications). 



20 The information on implementation of the model in middle schools in the district was gathered directly 
from Talent Development model developers in correspondence and conversations, unless otherwise noted. 

2 'Not counted in this sum is the “pilot” Talent Development middle school that adopted the model in 1995. As 
noted earlier, this school is not included in the current analysis due to data constraints and its status as a pilot test 

22 Data for the 2002-2003 and 2003-2004 school years were not available at the time of analysis. These fol- 
low-up years will be included in a report forthcoming in 2005. 
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The Talent Development Evaluation 
Figure 2 

Implementation Time Line in Six Early-Implementing Talent Development Middle Schools 
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NOTE: School B did not have a planning period because the model did not gain sufficient faculty approval until shortly before implementation began. 




























A unique aspect of the Talent Development model is the intense support that schools 
receive from CRESPAR-based curriculum developers and implementation staff and from 
school-based organizational facilitators. Both groups aim to ensure that the implementation of 
each major component receives two full years of intensive support, and they work closely with 
each site’s school leaders to adopt the model — being involved with everything from class 
scheduling and recruiting staff for new roles to ordering materials and coordinating professional 
development and coaching (described below). 

The relationship between the model developers and the school leaders is maintained 
throughout each year of implementation. Curriculum coaches at each site provide weekly im- 
plementation expertise and support while also serving as the liaison between school staff and 
the model developers. Although full implementation of the model should be feasible in a three- 
year time frame, CRESPAR has found that schools need four years to incorporate the core 
components. Even in sites that begin to implement all the components, fidelity and commitment 
to individual components may shift from year to year as teachers and administrators transfer out 
of the building and as new staff arrive. 

Implementation in Six Early-Implementing Middle Schools 

All the Talent Development middle schools in this analysis share certain components of 
the model, but the schools have varied somewhat in how and when they implemented the com- 
ponents. Some differences in implementation represent innovations in the model, and others 
reflect necessary flexibility when adapting the model to each unique school environment. As 
noted, the progress of implementation can be hampered by changes in school leadership as well 
as by turnover among teachers and staff. Given these factors, the schools varied greatly in terms 
of when each of the curricular programs was implemented, the use of available professional 
development, and the provision of an extra-help program. 

Reorganization of the School 

Every early-implementing middle school in the study reorganized in order to create 
supportive learning environments. 23 The schools were organized into small learning communi- 
ties, and interdisciplinary teams of teachers shared the same students. It is important to note that 
some of the very large middle schools (School C had over 1 ,300 students) have had greater dif- 
ficulty creating learning communities of 200 to 300 students and building close student-teacher 
relationships. 



23 Mac Iver et al., 2001. 
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Standards- Based Instructional Programs 

Figure 3 illustrates roughly when each of the four Talent Development instructional 
programs was formally introduced in the six early-implementing middle schools. Black blocks 
represent full initiation of a program; white blocks indicate that a program had not been started 
or was dropped; and striped blocks indicate partial initiation. For the purposes of this report, 
“full initiation” of an instructional component means that the school had purchased student and 
teaching materials, had agreed to send teachers to professional development, and had paid for 
coaching in that subject. However, teachers’ acceptance and willingness to use the materials as 
consistently and intensely as prescribed by the Talent Development model affected the thor- 
oughness of implementation. For example, “full initiation” does not indicate that all teachers 
participated in professional development around the curriculum (see the next section). For the 
purposes of this report, “partial initiation” means that the school adopted the instructional pro- 
gram for selected classrooms or grade levels. The evaluation was not able to collect data on the 
quality or fidelity of implementation. Nonetheless, Figure 3 shows that there was variation in 
the timing and completeness of adoption of the four standards-based instructional programs. It 
also shows that only two of the six schools had put all four programs in place for two or more 
years, although five schools had three or more of the programs in place. 

As noted above, all six of the early-implementing Talent Development middle schools 
used Student Team Literature as their Reading, English, and Language Arts (RELA) curricu- 
lum. 24 Most adopted it, at least partially, in the first year. All schools except School B had offi- 
cially adopted the reading program by the second year. School B never fully adopted the read- 
ing program in all classrooms and eventually dropped it in the fifth year of Talent Development 
implementation. 

Four of the six early-implementing Talent Development middle schools used the Uni- 
versity of Chicago School Mathematics Project (UCSMP) math program, which is recom- 
mended by Talent Development and includes algebra in eighth grade. The other two schools 
introduced other comparable standards-based math programs that were supported by Talent De- 
velopment. All six schools adopted the UCSMP program or another standards-based math pro- 
gram in all classrooms by the second year of Talent Development implementation, and they 
sustained the program throughout subsequent implementation years. 

Half of the six early-implementing middle schools adopted the science instructional 
program in some classrooms or grade levels in the first year. It typically took two to three years 
for schoolwide adoption of the science program. All six schools had the science program in 



Documentation of when the Talent Development middle school components were implemented was 
provided by Kathy Nelson, Talent Development Schools Education/Field Administrator (August 5, 2003). 
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The Talent Development Evaluation 
Figure 3 

Initiation of Curricular Components in Six Early-Implementing Talent Development Schools 
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NOTES: Implementation Years 1 to 5 are relative to the first calendar year of implementation in each school. 

For the purposes of this report, "full initiation" of an instructional component means that student and teaching materials were 
officially adopted in all classrooms at each grade level: The school purchased materials, agreed to send teachers to professional 
development, and paid for coaching in that subject. However, teacher acceptance and willingness to implement the materials 
affected the degree to which they were used. For the purposes of this report, "partial initiation” means that only some teachers 
at each grade level adopted the instructional program. Unfortunately, an indicator of the strength or fidelity of adoption is not 
available. 
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place by the third year of Talent Development implementation, though the breadth of use across 
grade levels varied from school to school. For example, in half the schools, the program was 
broadly adopted across all grade levels; in the other schools, some teachers at each grade level 
did not commit to using the science program. 

The U.S. History curriculum tended to be added in the second or third year of Talent 
Development implementation, if at all. It was fully initiated in three of the six early- 
implementing middle schools and partially initiated in two other schools. Student and teaching 
materials and ongoing professional development were provided to support the implementation 
of all four curricular initiatives. 

Support for Teachers 

As part of Talent Development’s professional development program, each of the six 
early-implementing middle schools in the study was assigned subject-specific curricular 
coaches. These coaches were either CRESPAR-based staff or teachers from the district who had 
been placed on special assignment to CRESPAR. Coaches were trained to provide weekly in- 
classroom support tailored to the needs of each teacher. For the most part, coaches were invited 
into classrooms to plan and model lessons, but coaches also observed teachers and gave confi- 
dential feedback and helped to adapt instructional programs to fit specific classroom needs. In 
addition, coaches and lead teachers from each school worked with CRESPAR-based instruc- 
tional facilitators to provide monthly workshops for each subject area. Teachers could also take 
part in other subject-specific professional development sessions outside the school day (in the 
afternoons, on weekends, or during the summer). Most schools were able to provide stipends to 
teachers who undertook this training. But at least one school was not able to mandate participa- 
tion in professional development activities, and no teachers from that school attended training 
during the three years of Talent Development implementation. 

Ongoing professional development varied somewhat by subject area. Professional de- 
velopment for math and science was grade-specific and unit-focused. Teacher and coaches dis- 
cussed specific units and modeled lessons that they used in class. Content materials helped to 
guide teachers’ implementation of the math and science programs to a significant extent. Pro- 
fessional development in history was grade-specific and focused on particular volumes in the 
10-volume series A History of US that teachers were using. Professional development for read- 
ing, English, and language arts (RELA) was somewhat more generic, as teachers within the 
same school often selected different novels. The RELA training was focused less on the novel 
itself and more on using partner discussion guides and developing lessons to engage and chal- 
lenge students. In this way, implementation of the English and language curriculum was heavily 
dependent on teachers’ adopting new instructional practices. 
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Extra Help 

Another component of the Talent Development Middle School model is the elective- 
replacement approach to extra help, whereby students receive extra help linked to the curricu- 
lum during the regular school day (replacing an elective or study hall for part of the year). This 
component is designed to help struggling students reach grade-level work in reading and math. 
In theory, the extra-help programs also support the model’s detracking approach, because 
teachers feel less pressure to “slow down” when there is a way for students to get more time and 
more instruction if needed. CRESPAR pilot-tested its extra-help reading program during the 
2000-2001 school year, while the extra-help math program was developed several years earlier. 
However, the schools are encouraged, not required, to offer extra help in reading and math as 
part of their elective offerings. Of the six early-implementing middle schools, only Schools C, 
D, and E had extra-help programs in place for one year or longer. 

Implementation in Five Later-Implementing Middle Schools 

In the first year of implementing the Talent Development Middle School model, four of 
the five schools that began Talent Development in the 2001-2002 school year had organized 
into small learning communities with teaching teams. Of these, three schools had adopted stan- 
dards-based reading and math programs; one school had adopted only the reading program but 
also had extra-help programs in both reading and math. The fifth school had adopted only the 
Talent Development standards-based science program. All five of these later-implementing 
schools had subject-specific curricular coaches, and ongoing professional development was 
available to support the curricular changes. 

Implications for the Impact Analysis 

The information in Figures 2 and 3 and the phase-in implementation strategy under- 
taken by the middle schools participating in the study have several implications for the focus 
and interpretation of the impact analysis discussed below. 

• The six early-implementing schools are the focus of the analysis. 

Figure 2 illustrates that the start of Talent Development implementation was staggered 
across middle schools in the district: Some schools began implementation earlier than others. 
All six of the early-implementing schools shown in the figure had at least three years of imple- 
mentation experience; two schools had four years of experience, and two others had imple- 
mented Talent Development for five years. Figure 3 shows that, within each school, implemen- 
tation of the four instructional programs were phased in over several years. Complete imple- 
mentation of the model is a three-year plan and usually a four-year reality. At this point in the 
project, it may be too early to expect significant impacts on student achievement and atten- 
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dance. However, such impacts are most likely to be found in schools that have implemented the 
model more fully. Therefore, the impact analysis primarily focuses on the six early- 
implementing schools that have worked with Talent Development for three, four, or five years. 
The five later-implementing schools had only one year of implementation experience, and their 
impact analysis is discussed as it compares with the analysis of the early-implementing schools. 

• The findings for the first three follow-up years are the most reliable. 

Follow-up data for the six early-implementing schools are available only for the first 
three years of Talent Development’s implementation. Because the analysis includes all six 
schools, findings for these three years will have the greatest statistical power and may also pro- 
vide the most robust indication of Talent Development’s preliminary impacts on student per- 
formance and engagement. Follow-up data are available for the fourth year of implementation 
at four schools and for the fifth year of implementation at two schools. The additional data pro- 
vide valuable information regarding the impacts of a more fully implemented Talent Develop- 
ment model and what impacts might be expected as the model matures in later years. However, 
because the pool of schools is smaller, average outcomes for these later years may be more eas- 
ily driven by one school. To identify such instances, the analysis considers year-by-year out- 
comes for each school. 

• The test score analysis focuses on eighth-grade math and reading. 

Given that the English and language arts and mathematics curricular components were 
typically the first to be implemented, achievement gains might be expected in these subjects 
first. The components were usually phased in by grade level, so it may have taken two or three 
years before a curricular component was adopted schoolwide. Given this implementation pat- 
tern, it is reasonable to expect that potential impacts on math and reading may precede gains in 
other subjects. However, given that some schools needed two or three years to put the reading 
and math components fully in place, it is also reasonable to expect that impacts, if any, during 
the first years of implementation may be more modest. The locus of impacts may be on eighth- 
grade students, who have benefited from two or three years of instructional improvements. For 
these reasons, the test score analysis focuses first on eighth-grade math and reading. Results for 
seventh-grade students are also included in the findings, although significant impacts may not 
be likely within the time frame of this analysis. 

• Impacts on mathematics may emerge during the current implementa- 
tion period. 

Also of note is the nature of professional development in math and reading. Profes- 
sional development in math is unit-focused and closely linked to the materials. This may enable 
teachers to readily transfer new instructional practices to the classroom. Professional develop- 
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ment in reading is more generic and is not necessarily tied to the current novel selected by each 
teacher. Because mastery of the new instructional practice in reading may take more time, 
change in this subject area may be significantly hampered by teacher turnover and by the influx 
of new teachers, who must master both content and pedagogy. 25 If this is the case, it may be rea- 
sonable to expect greater traction in math within three to four years, while results in reading 
may become evident in later years of implementation. 

• Greater personalization of the schools may lead to impacts on student 
attendance and promotion. 

Creating supportive learning environments by reorganizing the school into small learn- 
ing communities and teaching teams is also part of the first years of Talent Development im- 
plementation. The goal is to create a more communal organization of the school that personal- 
izes adult-child relationships, which leads to greater student engagement and more positive atti- 
tudes about school. Effects of this change may be seen in improved rates of attendance and 
grade-level promotion. The analysis focuses on these outcome measures over time — especially 
for eighth-grade students, as more students are exposed to supportive learning environments for 
two or more years. It is important to note that teacher and school leadership turnover may ham- 
per the evolution of more personalize learning environments, so that year-to-year change may 
not be cumulative. 



The Study’s Analytic Approach and Data Sources 

In order to determine the net effect of Talent Development, it is necessary to compare 
the experiences of a group of students who were exposed to the model with a truly comparable 
group of students who were not. The ideal research situation would provide an absolutely reli- 
able estimate of the student performance levels that would have been observed in the absence of 
the intervention (that is, a counterfactual ) and comparison of this estimate with actual student 
performance. Random assignment is the most reliable basis from which to construct estimates 
of the counterfactual, but, in this evaluation, it was not possible to randomly assign schools or 
students. Given this, the analytic approach that is used attempts to construct the best counterfac- 
tual possible in order to estimate the true impact of Talent Development. 

In this report, impacts are measured using a comparative interrupted time series design 
(see Box 2). 26 This analytic approach combines the use of the interrupted time series analytic 

~ 5 Useem found that — compared with new teachers in math, science, and social studies — new RELA 
teachers in Talent Development middle schools in the district were most likely to be poorly prepared to teach in 
their subject area and had little or no courses or pedagogy coursework in their content area (2001a, p. 15). 

26 A detailed description of the analytic approach is available online in this report’s Technical Resources 
(Unit 1: Analytic Appendix). See www.mdrc.org/pubhcations/400/techresources.pdf. 
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strategy with the use of the comparison schools analytic strategy to build on the strengths of 
each approach and to address the potential limitations of both. The comparative interrupted time 
series design compares deviations from the historical patterns for the Talent Development mid- 
dle schools with deviations from the historical patterns for similar non-Talent Development 
middle schools during the same period. Thus, impacts are defined as differences between Talent 
Development and non-Talent Development middle schools in terms of their deviations from 
historical patterns in student outcomes. When combined with regression analysis to control for 
differences caused by individual student background characteristics and prior school experi- 
ences, the approach seeks to isolate Talent Development’s unique impact on student engage- 
ment and performance. 



Box 2 

The Three Steps for Estimating Impacts 
with a Comparative Interrupted Time Series Design 

• Step 1: Estimating deviations from baseline in Talent Development schools. 

For each outcome under study, in each Talent Development school, the outcome 
level is compared with the pattern in the same school before it implemented the re- 
form (this is referred to in the report as the “baseline average”). 

• Step 2: Estimating deviations from baseline in non-Talent Development 
comparison schools. For each Talent Development school, the outcome levels in 
a group of comparison schools — a set of schools in the same district with charac- 
teristics similar to those of the Talent Development school — are compared with 
the baseline averages in these schools before the Talent Development school im- 
plemented the reform. 

• Step 3: Estimating the impact of Talent Development. Differences between the 
deviations from the baseline averages in the Talent Development schools and the 
deviations from the baseline averages in the non-Talent Development comparison 
schools are used to estimate the reform’s impact. 



The Interrupted Time Series Methodology 

The interrupted time series component of the analytic strategy assesses the extent to 
which measures of engagement and performance for students in Talent Development middle 
schools differ from the engagement and performance for similar students in the same schools prior 
to Talent Development implementation. This provides an indication of whether the participating 
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middle schools experienced a deviation from their historical patterns in student outcomes that was 
coincident with the introduction of Talent Development (the ‘‘interruption” in the interrupted time 
series design). The projection of each school’s recent history acts as the counterfactual. This is a 
particularly good counterfactual because, in the absence of the reform, many aspects of the school 
would be expected to stay the same (for example, students, faculty, school culture, neighborhood, 
and physical plant). The use of a historical pattern as the counterfactual has the potential to control 
for both measurable and unmeasurable characteristics of a school. 

However, the deviation from the baseline alone may not necessarily reflect the impact 
of Talent Development. Similar deviations from historical patterns could have been caused by 
districtwide policies or interventions that occurred at about the same time as Talent Develop- 
ment implementation. For example, while Talent Development was being scaled up, the district 
participated in an NSF Urban Systemic Initiative for mathematics. Such a change may cause 
positive deviations from baseline averages of math achievement in the district. An interrupted 
time series design would capture this improvement and would ascribe it to Talent Development; 
in reality, however, Talent Development may have caused some, all, or none of this change in 
math achievement. In order to sort out what part of the deviation from baseline is caused by 
Talent Development, this study looks at similar middle schools in the same district. 

The Comparison Schools Methodology 

The use of non-Talent Development comparison schools helps to account for other fac- 
tors in the broader school district that may influence school functioning and student engagement 
and performance. Each Talent Development school is matched with a set of non-Talent Devel- 
opment comparison schools that are similar on several dimensions. All Talent Development and 
comparison schools are nonselective, comprehensive middle schools in a single school district. 
The schools are matched in terms of racial/ethnic composition and average eighth-grade math 
and reading achievement scores. Table 3 provides an indication of the extent to which the 
matching process resulted in a group of non-Talent Development schools that is comparable to 
the 1 1 Talent Development schools in the study. In general, the table indicates that the non- 
Talent Development schools were similar to the Talent Development schools in terms of 
race/ethnicity and test scores over the years leading up to Talent Development implementation. 
Talent Development schools had a lower percentage of student who were overage for grade 
than their non-Talent Development counterparts. Comparing Table 3 and the eighth-grade col- 
umn in Table 2 shows that Talent Development schools and their comparison schools are repre- 
sentative of the 38 nonselective middle schools in the district in terms of average test scores in 
math and reading, attendance rates, and promotion rates. 
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The Talent Development Evaluation 
Table 3 



Characteristics of Eighth-Grade Students in Talent Development Schools 
and Non-Talent Development Comparison Schools, 

Averaged Over the Pre-Talent Development Baseline Period 



Characteristic 


Talent 

Development 

Schools 


Non-Talent 

Development 

Schools 


Difference 


Race/ethnicity (%) 


Black 


81.5 


81.8 


-0.3 


White 


4.4 


2.2 


2.1 


Hispanic 


11.2 


14.2 


-3.0 


Other 


2.9 


1.7 


1.2 


Overage for grade 2 (%) 


21.4 


23.8 


-2.4 


SSA test scores 


Average math and reading (NCE) 


27.0 


26.9 


0.1 


Math 


Average NCE 


25.9 


25.7 


0.2 


In the bottom quartile (%) 


77.5 


77.1 


0.4 


At or above grade level (%) 


6.3 


6.0 


0.3 


Reading 


Average NCE 


28.4 


28.5 


-0.1 


In the bottom quartile (%) 


71.1 


70.4 


0.7 


At or above grade level (%) 


9.0 


9.0 


-0.1 


Attendance rate b 


84.3 


84.8 


-0.5 


Attendance rate of 90% or higher 6 (%) 


47.3 


48.6 


-1.3 


Attendance rate of 80% or lower 6 (%) 


28.5 


26.6 


1.9 


Promoted to 9th grade 2 (%) 


97.5 


97.9 


-0.4 



(continued) 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 1 1 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

Results in the non-Talent Development columns reflect averages across 1 1 clusters, including both early-implementing 
and later-implementing school clusters, of non-Talent Development schools. Each cluster consisted of 1 to 12 non-Talent 
Development schools. Some non-Talent Development schools were counted in more than one cluster. 

Estimates are not regression-adjusted for students' background characteristics or prior achievement. 

Numbers reflect averages over the three-year period prior to the initial implementation of Talent Development for a 
given cluster. 
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Table 3 (continued) 



“Typically, students who were overage for grade were retained in the current grade or a prior one. “Overage for grade” 
means a student turned 12 before the start of the 6th grade, 13 before the start of the 7th grade, or 14 before the start of the 
8th grade. 

b Attendance rates were calculated for each student by dividing the number of days the student was present by the total 
number of days the student was enrolled in a given school year. 

“For the purposes of this analysis, 8th-grade students were considered to have been promoted to the 9th grade if they were 
listed as 9th-graders in the district's administrative data file one year after the current year. Students whose records were 
not included in the data file one year after the current year, for whatever reason, were not in the analysis sample for this 
outcome. 



Measures of student achievement and engagement at the comparison schools should be 
a good estimate of what might have been observed in Talent Development schools in the ab- 
sence of the intervention; that is, they should be a good counterfactual. But differences between 
the Talent Development and comparison schools do not necessarily reflect only the impact of 
Talent Development. Some differences could be artifacts of differences in the prior trends in 
student engagement and performance. For example, suppose that test scores for students in Tal- 
ent Development schools were lower than test scores for students in non-Talent Development 
schools and that the scores improved only marginally after Talent Development began. At the 
same time, however, test scores for students attending similar schools in the district may actu- 
ally have been declining over the same period. In this instance, Talent Development may have 
had a positive impact by preventing test scores from dropping, rather than by improving the 
overall average. Such a pattern could be observed only by comparing an interrupted time series 
for both Talent Development and non-Talent Development comparison schools, that is, by 
comparing changes over time in Talent Development schools with changes over time in non- 
Talent Development comparison schools. 

Controlling for Changes in School Composition 

This analysis takes into account the fact that Talent Development schools (or non- 
Talent Development comparison schools) may experience a change in the composition of their 
student populations. For example, neighborhoods may undergo demographic changes or 
changes in geographic boundaries or rules governing school assignment. More important, Tal- 
ent Development may cause a change in the student population by, for example, increasing at- 
tendance and reducing mobility and perhaps keeping lower-performing students in school 
longer. In order to help account for systematic changes in the characteristics of student cohorts 
over time, the analysis incorporates individual student characteristics into the analytic model. 
Specifically, the analytic model includes controls for race/ethnicity, fourth-grade test scores in 
reading and mathematics, and whether the student has repeated a grade. 
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Analytic Assumptions 

The goal of the comparative interrupted time series approach is to make the causal in- 
ference that Talent Development produced the observed changes, if any, in student outcomes. In 
other words, the analytic approach attempts to distill the impact of Talent Development from 
other factors that may affect student outcomes. In order to make this causal inference, several 
assumptions are made. Box 3 outlines these assumptions, and this section explores the assump- 
tions and provides some context for their validity. 



Box 3 

Analytic Assumptions of the 
Comparative Interrupted Time Series Design 

• The projected baseline average for an outcome is a reliable indicator of a given school’s 
future performance in the absence of an intervention like Talent Development or an 
event aimed at changing that outcome. 

• Schools with characteristics similar to Talent Development schools provide a reliable 
indicator of how student outcomes are likely to respond to districtwide policies or 
events during the Talent Development implementation period. 

• Background characteristics of students enrolled before Talent Development implemen- 
tation are the same as those of students enrolled in Talent Development and comparison 
schools during implementation of the model (or statistical controls adequately account 
for such differences). 

• The process by which schools enter into the Talent Development network does not af- 
fect student outcomes. 



First, the causal inference is based, in part, on the assumption that the projected baseline 
average for an outcome is a reliable indicator of a given school’s future performance in the ab- 
sence of an intervention like Talent Development or an event aimed at changing that outcome. 
Year-to-year stability of most outcomes in the baseline period provides some confidence that 
this is a reasonable assumption to make for the analysis. Neither Talent Development nor non- 
Talent Development schools displayed substantial positive or negative trends prior to the im- 
plementation of the model. 

Second, the causal inference is based, in part, on the assumption that schools with char- 
acteristics similar to Talent Development schools provide a reliable indicator of how student 
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outcomes are likely to respond to districtwide policies or events during the Talent Development 
implementation period. Both the comparability of the characteristics of Talent Development and 
non-Talent Development schools and the year-to-year stability of baseline student outcomes in 
both sets of schools suggest that this is a reasonable assumption to make. Table 3 shows that 
Talent Development schools and their comparison schools served similar students with similar 
outcome levels in the baseline period. Further, these outcomes levels did not change signifi- 
cantly during the pre-Talent Development baseline period. One challenge to this assumption is 
that one or two comparison schools experienced significant and atypical improvement or de- 
cline in the follow-up period. In fact, the analysis looked for schools that appeared to be outliers 
in the follow-up period; there were no major outliers that skewed the means of the comparison 
schools. Thus, there is even greater confidence in the reliability of the comparison schools. 

Third, a necessary assumption is that background characteristics of students who en- 
rolled before Talent Development implementation are the same as those of students who en- 
rolled in Talent Development schools and comparison schools during implementation of the 
model. One competing hypothesis that might explain changes in student outcomes — other than 
an effective intervention — is that the composition of the student body changed from the base- 
line to the follow-up period. For example, the introduction of a magnet program may attract 
more able or more motivated students to a school and thus raise the school’s average test scores. 
This analysis assumes that variables included in multiple regression adequately control for 
compositional changes in student characteristics. The analysis accounts for shifts in racial/ethnic 
composition and changes in levels of prior achievement (using fourth-grade test scores), as well 
whether students have repeated a prior grade. There may be changes in other student character- 
istics that correlate with student outcomes — like student motivation — that are not included in 
the regression model because the data are unavailable or cannot be quantified. 

Even if these assumptions are valid, there may still be alternative explanations or other 
factors unrelated to Talent Development that contribute to the observed differences in student 
outcomes. For example, the analysis does not account for the process by which schools enter 
into the Talent Development network. Some may argue that schools with more entrepreneurial 
leaders — who are more likely to seek out a reform model like Talent Development — may 
experience improved student outcomes even in the absence of the intervention. The analysis is 
unable to rule out this possibility. Despite the limitations of the comparative interrupted time 
series approach, it offers a valid estimate of the impact of Talent Development in middle 
schools in the district, particularly when estimates are pooled across several schools. 

Pooling Across Schools 

The pooled estimates across the six early-implementing middle schools that are the 
primary focus of this study maximize the reliability of the impact estimates. By pooling esti- 
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mates, the analysis can assess the likelihood that a nonzero impact results from chance. In gen- 
eral, the larger the number of schools that exhibit a nonzero impact, the higher the likelihood 
that the analysis can detect real changes in student engagement and performance that were pro- 
duced by Talent Development. Although the focus is on results from pooled estimates, results 
for individual schools are also discussed in order to highlight variation across sites; it should be 
noted, however, that these impacts are less reliable than the pooled estimates, because estimates 
for any one school may be anomalous. 

Follow-up data for all six early-implementing school clusters are available only for the 
first three years of implementation. Findings for these three years have the greatest statistical 
power and show the most robust indication of Talent Development’s preliminary impact on stu- 
dent performance and engagement. Also, indications of statistical significance, 27 which depend 
in part on sample size, may be achieved with impacts of a smaller magnitude in the first three 
years of implementation as compared with impacts in Years 4 and 5, which include fewer 
schools. Similarly, average deviations from baseline for Talent Development schools include 6 
schools at most, compared with average deviations from baseline that include up to 18 non- 
Talent Development schools. Again, in this instance, statistical significance may be achieved 
with smaller deviations from baseline for non-Talent Development comparison schools as com- 
pared with deviations from baseline for Talent Development schools. 

Data Sources 

The primary sources of data for the impact analysis are individual students’ school re- 
cords, which were obtained from the district. In general, administrative, attendance-related, and 
course-related information was obtained for all middle and high school students in the district at 
the end of each school year, from 1995-1996 through 2001-2002. 28 Box 4 defines several key 
outcomes included in the analysis. 



27 Statistical significance is a measure of the degree of certainty that some nonzero deviation from the 
baseline average actually occurred. For example, if an impact estimate is statistically significant, then one may 
conclude with some confidence that the program really had an effect. If an impact estimate is not statistically 
significant, then the nonzero estimate is more likely to be the product of chance or random variation in the 
averages that were calculated across the schools and years under study. Unless otherwise noted, the deviations 
from baseline averages and the Talent Development impacts discussed in this report are statistically significant 
at the 10 percent level or less; that is, there is no more than a 10 percent probability that the difference results 
only from chance or random variation. 

28 This report’s online Analytic Appendix (Unit 1 of the Technical Resources) gives a brief overview of the 
types of information included in these data sets. See www.mdrc.org/publications/400/techresources.pdf. 
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Box 4 



Definitions of Key Program Outcomes 

Test Scores 

• State Standards Assessment (SSA): A criterion-referenced test that provides in- 
formation about student performance on skills and content knowledge specified by 
the state. This test is given in eighth grade in the district. 

• Stanford Achievement Test, Version 9 (SAT-9): A national norm-referenced 
test that measures individual student achievement relative to scores obtained by a 
random sample of students from across the country. This test is given in seventh 
grade in the district. 

• Metrics for the SSA and SAT-9 

• Normal Curve Equivalent (NCE) score: The average NCE score for stu- 
dents taking the test in a given subject area. The normalized test score, 
which ranges from 1 to 99 with a mean of 50, allows for comparisons across 
tests and subjects. The norms were constructed based on a statewide sample 
of students for the SSA and a national sample for the SAT-9. 

• At or above grade level: The percentage of students scoring at or above grade 
level on the test as indicated by scoring at or above the 50th percentile. 

• In the bottom quartile: The percentage of students scoring at or below the 
25th percentile on the test. 

Attendance 

• Attendance rate: The total number of days that a student was marked as present 
during a school year, divided by the total number of days that the student was 
listed as enrolled. (These data are consistently available only for students who at- 
tended school for at least one day in the fourth marking period of the school year - . 
Thus, the analysis is not able to include students who dropped out or left the dis- 
trict before that point.) 

• Regular attendance: An attendance rate of 90 percent or higher for the year. 

• Chronic absenteeism: An attendance rate of 80 percent or lower for the year. 

Promotion 

• Promoted to ninth grade: The classification for a student who was designated in 
the district's administrative records as an eighth-grader in a given school year and 
was designated as a ninth-grader in the following school year. Students who are 
not in the district's administrative records in either year were not classified. 
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Preliminary Impact Findings 

The preliminary impact findings for this study indicate that, for eighth-grade students in 
early-implementing schools in the district, Talent Development produced statistically significant 
gains in math achievement that strengthened during the implementation period. The model also 
had a modest impact on attendance outcomes for eighth-grade students. The model did not pro- 
duce consistent impacts, positive or negative, on reading achievement for eighth-grade students. 
The findings do not show significant impacts on seventh-grade math and reading achievement, 
though Talent Development middle schools show some improvement in the later years of im- 
plementation. Attendance rates for seventh-grade students did not have a consistent pattern of 
change during the implementation period. 

The findings in this section are presented in two ways. First, summary tables show year- 
by-year average impacts for several outcomes, along with corresponding effect sizes. Second, 
bar graphs show the deviation from baseline average for both Talent Development and non- 
Talent Development schools. Impact estimates — which are featured in the summary tables and 
also indicated on the bar graphs — are defined as the difference between deviations from base- 
line for Talent Development schools and deviations from baseline for non-Talent Development 
schools. The year-by-year tables and figures show the pattern of impacts over time that may 
relate to the phased-in implementation of the model. Both the tables and the figures show aver- 
age impact estimates across the six middle schools in the district that had implemented Talent 
Development for three, four, or five years — the ones that this report calls “early-implementing 
schools.” Because only two schools have implemented for five years and four schools have im- 
plemented for four years, aggregate impacts for Years 4 and 5 are not based on averages for all 
six schools. Impacts in these last two years should be interpreted with more caution. In addition, 
the report’s Technical Resources provide estimates for each school by year. 29 These data help to 
reveal patterns, but estimates for any one school are not reliable. 

As of the 2001-2002 school year, all six of the early-implementing schools had adopted 
Talent Development’s reading and math programs and had established small learning commu- 
nities and teaching teams. The discussion focuses on impact findings in reading and math 
achievement and on attendance and promotion outcomes for seventh- and eighth-grade students 
in these six schools. Five other middle schools in the district began implementing Talent Devel- 
opment in the 2001-2002 school year. Since only one year of data is available for these later- 
implementing schools, their findings are based on a separate analysis and are discussed as they 
compare with the findings for the early-implementing schools. Because the analysis in this re- 
port includes a limited number of schools — some that have only a few years of follow-up data 

29 For school-by-year tables for each outcome, see www.mdrc.org/publications/400/techresources.pdf. 



- 34 - 



and histories of only partial implementation — these findings are considered preliminary. A 
forthcoming report will include two additional years of data. 



Impacts for Eighth-Grade Students 

The analysis first focuses on the eighth grade, for two reasons. First, eighth grade marks 
the culmination of the middle school experience for students and the start of a critical transition 
period for young people. In many ways, the engagement and performance of eighth-grade stu- 
dents reflects an accumulation of the instruction and support that they received in middle 
school. Also, their engagement and performance are critical indicators of their preparation to 
meet the challenges of making a successful transition to high school. Second, the Talent Devel- 
opment Middle School model makes an effort to provide additional supports and to upgrade 
curricula and instruction for all middle school grades. Thus, the impact of Talent Development 
on the engagement and performance of eighth-grade students, in many ways, represents cumu- 
lative effects of the model for middle schools. If Talent Development does make a significant 
difference for middle schools, it is most likely to be evident among eighth-grade students, who 
had the potential to receive multiple years of its treatment. 30 Because the analysis does not con- 
trol for seventh-grade achievement, eighth-grade findings represent the cumulative impact of 
Talent Development. 

A particularly important outcome is student achievement. Eighth-grade students in the 
district take State Standards Assessment (SSA) tests in both math and reading. This has become 
a high-stakes assessment in the district. In the pre-Talent Development baseline period, students 
in both Talent Development and non-Talent Development middle schools scored far below the 
state averages on these tests. The baseline averages for these schools were at about the 24th 
Normal Curve Equivalent (NCE) in math and at about the 28th NCE in reading. The SSA 
statewide average for both tests is the 50th NCE. 

• Talent Development had a positive impact on math achievement of 
eighth-grade students that strengthened over time. The findings indicate 
that Talent Development raised the average NCE score and significantly 
reduced the percentage of students scoring in the bottom quartile of the 
statewide assessment in mathematics. 



30 It is important to note that the eighth-grade class in a given middle school is likely to include a number 
of students who have attended that school for only one or two years. It may be argued that impact estimates 
based on the full sample of eighth-grade students may reflect a diluted version of the Talent Development 
treatment, because such students would have received only part of the full treatment. These estimates do, how- 
ever, reflect the reality in which Talent Development operates, and it is unlikely that the model will ever be 
implemented in a situation where student mobility is prohibited. 
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Figure 4 illustrates the trend in math achievement over the first three years of imple- 
mentation, averaged across all six early-implementing schools. The figure shows the deviation 
from baseline in math achievement, as measured by average NCE score, for Talent Develop- 
ment and non-Talent Development schools over time. Both sets of schools improved during the 
follow-up period, but Talent Development schools outpaced their non-Talent Development 
comparison schools. For example, in Year 3, Talent Development schools improved from their 
baseline average by about 6 NCEs, while non-Talent Development schools improved from their 
baseline average by about 4 NCEs. The difference in the deviations from baseline for the two 
sets of schools is the estimated impact of Talent Development in each follow-up year. There- 
fore, in Year 3, the estimated impact of Talent Development on math achievement is 2 NCEs. 31 
Though the magnitude of this impact may seem small, it represents about one-sixth of the stu- 
dent-level standard deviation for this outcome, or an effect size of 0.1 5. 32 A recent study notes 
that a full year of classroom instruction (albeit calculated for fifth grade, not eighth grade) has 
been associated with one-half the student-level standard deviation in math achievement — or an 
effect size of about 0.50 — as measured by a nationally normed test. 33 Therefore, impacts from 
a comprehensive school reform model like Talent Development, in the range of a 0.15 effect 
size, may be considered educationally important. 

As noted above, all six early-implementing schools have at least three years of follow- 
up data. Four years of follow-up data are available for four schools, and five years of follow-up 
data are available for two schools. Since impact estimates for Years 4 and 5 are based on fewer 
schools, they should be interpreted with greater caution. However, impact estimates for these 
years may provide a good indication of Talent Development’s impacts as the model matures, if 
the pattern of impacts for Years 1 to 3 are similar across each cohort of schools; that is, if the 
pattern of impacts for the first schools to implement the model is not very different from the 
pattern of impacts for all schools. 



3 'it is important to note that, had the analysis simply looked at the average deviation from baseline for Tal- 
ent Development schools, the impact of Talent Development would have been overestimated to be 5.7 NCEs. 
Also, had the analysis simply compared the follow-up period averages for Talent Development and non-Talent 
Development schools, the impact of Talent Development would have been underestimated to be 1.4 NCEs (the 
difference between 29.2 NCEs and 27.8 NCEs, shown in Appendix Table A.l). 

32 Effect sizes show each impact as a proportion of the comparison group student-level standard deviation 
for each outcome. For example, an impact of 2.2 NCEs in SSA math test scores corresponds to an effect size of 
0.15, or about one-sixth, of the student-level standard deviation for this outcome in the pre-Talent Development 
period. 

33 Kane (2004) notes that, in the national samples used to norm the SAT-9 scores, students taking the test 
in the spring of fifth grade scored approximately one-third of a standard deviation higher in reading and one- 
half of a standard deviation higher in math than students taking the test in the spring of fourth grade. 
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The Talent Development Evaluation 
Figure 4 



Impacts on SSA Math NCE Scores 

for Eighth-Grade Students in Early-Implementing Talent Development Schools, 
Three-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

Results are pooled over six Talent Development Schools and six clusters of non-Talent Development comparison 
schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in math NCE points from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development 
schools and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 



The top panel of Figure 5 shows the trend in math achievement over the first three years 
of implementation, averaged across all six early-implementing schools (the same as in Figure 
4); the middle panel shows the trend in math achievement over the first four years of implemen- 
tation, averaged across four of the six early-implementing schools; and the bottom panel shows 
the trend in math achievement, averaged across two of these four early-implementing schools. 
While there is some variation, the patterns of impacts in the first three years of follow-up are 
similar from panel to panel. Impacts in math achievement tend to improve over time and seem 
to strengthen in Years 3 and 4. The bottom panel, which includes only two schools, shows a 
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The Talent Development Evaluation 
Figure 5 

Pattern of Impacts on SSA Math NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools, 
for School Clusters with Varying Years of Implementation 
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All six school clusters had at least three years of implementation. 
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Four of the six school clusters had at least four years of implementation. 

Imp act = 3.9 *** 

Impact = 2.1 

Impact = -0.47 ^ 

Impact =1.3 | | i 

ft in n 




Year 1 



Year 2 



Year 3 



Year 4 



■ Talent Development (Schools A-D) 
Follow-Up Year □ Non-Talent Development 



Two of those four school clusters had five years of implementation. 

Impac t = 5.0 ** Impact = 4. i * 

Impact = 2.7 



Impact = -0.2 



Impact = -0.2 

£ 




Year 1 



Year 2 Year 3 Year 4 Year 5 

■ Talent Development (Schools A-B ) 

Follow-Up Year □ Non-Talent Development 

(continued) 



- 38 - 





Figure 5 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

Panel I: Results are pooled over six Talent Development schools and six clusters of non-Talent Development comparison 
schools, school clusters A, B, C, D, E, and F. 

Panel II: Results are pooled over four Talent Development schools and four clusters of non-Talent Development 
comparison schools, school clusters A, B, C, and D. 

Panel III: Results are pooled over two Talent Development schools and two clusters of non-Talent Development 
comparison schools, school clusters A and B . 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in math NCE points from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development schools 
and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends 
in part on sample size, may be achieved with impacts of a smaller magnitude in the top panel, as compared with the middle 
and bottom panels, which include fewer schools. 



slight dip in Year 5, but this is still a strong impact. In Years 4 and 5, gains for Talent Develop- 
ment schools were about twice those of non-Talent Development comparison schools, with im- 
pacts of about 4 NCEs in each year — which corresponds to an effect size of 0.28. 

Table 4 summarizes the impact findings for several measures of engagement and per- 
formance for eighth-grade students. Although the sample changes over time, for simplicity, im- 
pact estimates for Years 4 and 5 are included with those for Years 1 to 3 in this table and subse- 
quent exhibits. The first three outcomes in Table 4 relate to SSA math test scores. While the 
average NCE score provides a sense of how students measure up against statewide norms, an- 
other important test score outcome is the percentage of students scoring in the bottom quartile. 
As discussed above, Talent Development aims to work with low-performing schools. In fact, in 
the baseline period, about 80 percent of students in the Talent Development schools and the 
non-Talent Development comparison schools scored in the bottom quartile in math. In Figure 6, 
each bar represents a reduction in the percentage of students scoring in the bottom quartile. Both 
sets of schools reduced this percentage, but Talent Development schools outpaced their com- 
parison schools by 3 to 12 percentage points. The largest impact is in Year 4. Talent Develop- 
ment schools reduced the percentage of students scoring in the bottom quartile by about 20 per- 
centage points, while non-Talent Development comparison schools reduced this percentage by 
about 8 percentage points. This impact represents an effect size of -0.30, which is almost one- 
third of the student-level standard deviation. 
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The Talent Development Evaluation 
Table 4 



Impacts on SSA Test Scores, Attendance, and Promotion 
for Eighth-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 







Impact at Follow-Up ' 1 








Impact Effect Size * 3 




Outcome 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


SSA test scores 


Number of school clusters 


6 


6 


6 


4 


2 


6 


6 


6 


4 


2 


Math 


Average NCE 


1.3 


0.6 


2.2 * 


3 9 *** 


4.1 * 


0.09 


0.04 


0.15 * 


q 27 *** 


0.28 * 


In the bottom quartile (%) 


- 2.8 


-4.7 


-4.4 


-12 1 *** 


-10.9 


-0.07 


- 0.12 


- 0.11 


-0.30 *** 


-0.28 


At or above grade level (%) 


- 0.6 


- 1.2 


0.9 


4.5 ** 


4.5 


-0.03 


-0.06 


0.04 


0.20 ** 


0.20 


Reading 


Average NCE 


- 1.1 


2 7** 


0.1 


0.7 


2.8 


-0.07 


0.17 


** 0.01 


0.05 


0.17 


In the bottom quartile (%) 


3.2 


-5.6 * 


0.5 


-3.3 


- 8.8 


0.07 


- 0.12 


* 0.01 


-0.07 


-0.19 


At or above grade level (%) 


0.0 


4.0 ** 


0.6 


1.8 


4.0 


0.00 


0.14 


** 0.02 


0.06 


0.14 


Attendance' 


Number of school clusters 


5 


5 


5 


4 


2 


6 


6 


6 


4 


2 


Attendance rate 


1.6 


2.1 * 


2.3 * 


-0.1 


-1.0 


0.05 


0.07 


* 0.08 * 


0.00 


-0.03 


Attendance rate of 90% or higher (%) 


6.3 


4.4 


9 -j ** 


4.0 


2.0 


0.13 


0.09 


0.20 ** 


0.08 


0.04 


Attendance rate of 80% or lower (%) 


-3.8 


-3.9 


-5.7 * 


-1.3 


2.5 


-0.08 


-0.08 


-0.12 * 


-0.03 


0.05 


Promotion 1 


Number of school clusters 


6 


6 


6 


4 


2 


6 


6 


6 


4 


2 


Promoted to 9th grade (%) 


0.0 


1.1 


2 5 *** 


3.1 ** 


0.4 


0.00 


0.05 


0 ii *** 


0.13 ** 


0.02 
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Table 4 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from the 6 early-implementing Talent Development middle schools and 18 non-Talent Development middle schools. The 
analysis sample includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days 
during a given school year. 

The number of school clusters varies by year due to the staggered implementation schedule and availability of data. Results for SSA test scores and promotion in Year 
1 , Year 2, and Year 3 are pooled over six Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 4 are pooled 
over four Talent Development schools and four clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent 
Development schools and two clusters of non-Talent Development comparison schools. 

Attendance measures were only available for five clusters. Therefore, results for Year 1, Year 2, and Year 3 are pooled over five Talent Development schools and five 
clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four clusters of non-Talent 
Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent Development comparison 
schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had 
repeated a prior grade. 

“The impact at follow-up for a given year was calculated as the difference in deviations from the baseline average between early-implementing Talent Development 
schools and their non-Talent Development comparison schools. A two-tailed t-test was applied to the impact at follow-up. Standard errors were adjusted to account for 
cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends in part on sample 
size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with Year 4 and Year 5, which include fewer schools. 

b The impact effect size for each outcome was calculated by dividing the impact in a given year by the standard deviation of that outcome for all 8th-grade students in 
the 1 1 Talent Development schools and 18 non-Talent Development comparison schools from a pre-Talent Development period, school years 1995-1996 and 1996- 
1997. 

“Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a 
given school year. Attendance measures were not available for one Talent Development school. School E. 

d Eighth-grade students were considered promoted if they were listed as 9th-grade students in the district’s administrative data file one year after the current year. 
Students whose records were not included in the data file one year after the current year, for whatever reason, were not in the analysis sample for this outcome. 
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Figure 6 

Impacts on the Percentage of Students in the Bottom Quartile of SSA Math Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools, 
Five-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. 
Therefore, results for Year 1, Year 2, and Year 3 are pooled over six Talent Development schools and six clusters of non- 
Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development 
schools and two clusters of non-Talent Development comparison schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations front baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in the percentage of students with SSA math scores in the bottom quartile from the three-year pre-implementation baseline 
average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development 
schools and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends 
in part on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared 
with Year 4 and Year 5, which include fewer schools. 
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The presence of larger impacts in the later years of implementation is consistent with 
the phased-in implementation pattern of a standards-based math curriculum in Talent Develop- 
ment schools. It is also consistent with the potential cumulative effect of Talent Development, 
as discussed above. Eighth-grade students who enrolled during Years 3 through 5 of implemen- 
tation are likely to have been “exposed to” Talent Development in sixth and seventh grades as 
well as in eighth grade. These students — although not all students, given student mobility — 
have received a “full dose” of Talent Development in their middle school years. 

There is some variation in the magnitude and even in the direction of math impacts 
across the six early-implementing schools. 34 By Year 3, all the schools had positive impacts of 
between 2 and 3 points on average NCE scores. Year 4 impacts were generally stronger for 
each school, ranging between 2 and 8 NCE points, and these scores held relatively steady for 
the two schools for which a fifth year of data is available. The impact of Talent Development on 
math achievement in the five later-implementing schools, which had implemented the model for 
only one year, is also positive and of the same magnitude as the impact estimates for the first 
year of implementation in early-implementing schools. The upcoming report will examine 
whether these later-implementing schools are able to sustain and build on their first-year gains. 

• Talent Development did not produce consistent impacts on eighth-grade 
reading achievement over the follow-up period. 

Table 4 shows that the impacts on reading achievement do not follow a consistent pat- 
tern. First-year impacts, though small and not statistically significant, are negative, while some 
of the largest positive and statistically significant impacts in reading are found in Year 2. Figure 
7 shows that, in the second year of implementation, Talent Development schools improved av- 
erage reading scores by about 2 NCEs and that, at the same time, average reading scores de- 
clined by about 1 NCE in the comparison schools. Hence, the impact of Talent Development 
was an increase of about 3 NCEs in Year 2. This corresponds to an effect size of 0.17, which is 
not trivial when a full school year of instmction is associated with about one -third of the stu- 
dent-level standard deviation in nationally normed reading test scores. 35 However, this impact 
was not sustained over time. 

Talent Development and non-Talent Development schools both improved slightly in 
Year 3. The Talent Development schools built on this gain in Years 4 and 5, with increases of 
about 2 and 4 NCEs, respectively, while gains in the comparison schools remained at about 1 
NCE, on average. In fact, further analysis indicates that all but one Talent Development school 



34 For school-by-year impact tables, see www.mdrc.org/publications/400/techresources.pdf. 
35 Kane, 2004. 
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Figure 7 

Impacts on SSA Reading NCE Scores 

for Eighth-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. 
Therefore, results for Year 1, Year 2, and Year 3 are pooled over six Talent Development schools and six clusters of non- 
Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development 
schools and two clusters of non-Talent Development comparison schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in SSA reading NCE points from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development schools 
and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends 
in part on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared 
with Year 4 and Year 5, which include fewer schools. 
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had consistently small but positive deviations from the baseline average in reading, while there 
was greater variation in the size and direction of change in the comparison schools. However, it 
is not clear that findings for Year 5 provide a good indication of potentially greater impacts in 
reading, as they are driven by one school. 36 

A similar story emerges for the second reading outcome, the percentage of students 
scoring in the bottom quartile. In the baseline period, about 70 percent of students in both the 
Talent Development and the non-Talent Development comparison schools scored at this low 
level. However, there is greater variability from year to year and school to school. Again, the 
large impact in Year 5 is driven by one school that had particularly strong impacts throughout 
all five years of implementation. The impact of Talent Development on reading achievement in 
the first year of implementation in the five later-implementing schools was small but in the right 
direction, unlike the first-year impacts described above. 

The findings for math and reading achievement appear to be consistent with the features 
of implementation. Most Talent Development schools officially adopted the math and reading 
programs at the same time: A school purchased materials, agreed to send teachers to profes- 
sional development, and paid for coaching in those subjects. Professional development in math 
was unit-focused and grade-specific, which may have facilitated implementation more than the 
technique-focused training in reading and language arts. The reading program was more reliant 
on teachers’ accepting and mastering new instructional practice, which is expected to develop 
over time but may be less resilient to staff turnover. 

• During most years of follow-up, average eighth-grade attendance rates 
modestly improved in both Talent Development and non-Talent Devel- 
opment schools. In general, Talent Development schools outpaced their 
comparison schools. 

Table 4 contains three outcomes related to attendance for eighth-grade students. The 
first, attendance rate, is calculated by dividing the number of days present by the number of 
days enrolled for each student. Figure 8 shows that, during the follow-up period, the overall av- 
erage attendance rate improved in both Talent Development and non-Talent Development 
schools. This finding is consistent across all schools. Statistically significant differences, of 
about 2 percentage points, between Talent Development and non-Talent Developments schools 
are seen in Years 2 and 3. Overall attendance rates averaged between 85 percent and 88 percent 
in both sets of schools during the five years of follow-up. 



36 Note that the impact of 2.8 NCEs in Year 5 is not statistically significant, while the impact of 2.7 NCEs 
in Year 2 is significant at the 5 percent level. The sample of schools in Year 2 includes six clusters of Talent 
Development and comparison schools, while the sample of schools in Year 5 includes only two clusters. 
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Figure 8 

Impacts on Attendance Rates 

for Eighth-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 




SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. Attendance data were not 
available for one Talent Development school, School F. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. 
Therefore, results for Year 1, Year 2, and Year 3 are pooled over five Talent Development schools and five clusters of non- 
Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development 
schools and two clusters of non-Talent De velopment comparison schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in attendance rate from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development schools 
and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to account 
for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical 
significance, which depends in part on sample size, may be achieved with impacts of a smaller magnitude in the first three 
follow-up years, as compared with Year 4 and Year 5, which include fewer schools. 
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In the pre-Talent Development baseline period, about half the students in both Talent 
Development and non-Talent Development schools were regularly attending school — defined 
as having an attendance rate of 90 percent or higher. Talent Development puts a great deal of 
emphasis on student attendance, so it is not surprising to see that the percentage of regularly 
attending students rose during implementation of the model. Every Talent Development school 
had at least one year of double -digit gains on this outcome and, in each follow-up year, out- 
paced its comparison schools. 

There is a similar pattern of findings for lowering the percentage of chronic absentees 

— defined as having an attendance rate of 80 percent or lower — which included about 28 per- 
cent of students in the baseline period. In every follow-up year, Talent Development schools 
reduced the percentage of chronic absentees and, in Years 1 through 4, did so to a greater extent 
than their comparison schools. In Year 5, Talent Development schools reduced chronic absen- 
tees by about 4 percentage points, while non-Talent Development comparison schools reduced 
them by about 6 percentage points. 

Interestingly, these impacts are driven by three schools that began Talent Development 
implementation in 1998-1999 and 1999-2000, rather than the first two schools to implement the 
model. 17 The strength of the impacts drops off in Years 4 and 5, when fewer of these schools are 
part of the analysis. Although this finding suggests that attendance impacts improved for suc- 
cessive cohorts of Talent Development schools, findings for the five later-implementing schools 
do not confirm this pattern. In general, first-year attendance impacts for these schools are not 
stronger than first-year impacts for the early -implementing schools. 

• In both Talent Development and non-Talent Development schools, over 
95 percent of students were promoted from eighth to ninth grade. In one 
implementation year, the findings indicate that Talent Development sig- 
nificantly improved this promotion rate. 

The last outcome in Table 4 shows the percentage of eighth-grade students who were 
promoted to the ninth grade. Given the very high rates of promotion from eighth to ninth grade 

— which in the baseline period were nearly 98 percent for both Talent Development and non- 
Talent Development schools — a significant impact on this outcome was not expected. Indeed, 
the table shows small but positive impacts for each year of implementation. On average, Talent 
Development schools slightly improved promotion rates, compared with their baseline aver- 
ages, while non-Talent Development comparison schools had virtually no change or slight de- 
clines in promotion rates from eighth to ninth grade. In Year 3, Talent Development had a sta- 
tistically significant impact of nearly 3 percentage points on the promotion rate. 



37 For school-by-year impact tables, see www.mdrc.org/publications/400/techresources.pdf. 
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Impacts for Seventh-Grade Students 

Seventh-grade students benefited from, at most, two years of Talent Development. 
Thus, it is reasonable to expect that impacts for seventh-graders may be less pronounced than 
impacts for eighth-graders. And given the multiyear implementation of the model, impacts for 
seventh-grade students, too, are likely to be strongest in the later years of implementation. In 
fact, it may be too early to look for significant impacts on seventh-grade students. 

Table 5 provides a summary of the impact findings for several measures of engagement 
and performance for seventh-grade students. Again, although the sample changes over time, 
impact estimates for Years 4 and 5 are included with those for Years 1 through 3 in this table 
and in the exhibits that follow. Also, because the analysis does not control for sixth-grade 
achievement, seventh-grade findings represent the cumulative impact of Talent Development. 

Test scores available for the seventh grade are from the Stanford Achievement Test, 
Version 9 (SAT-9). Impact estimates are presented here for math total, math problem solving, 
and reading comprehension. In the pre -Talent Development baseline period, students in both the 
Talent Development and the non-Talent Development comparison schools scored below the 
national averages on these tests. The baseline average for these schools was at about the 34th 
NCE in math total and math problem solving and at about the 36th NCE in reading comprehen- 
sion. The national average for both tests is the 50th NCE. 

• Talent Development did not produce systematic impacts on seventh- 
grade math achievement in the first three years of implementation. 

There is some evidence, however, that positive impacts began to emerge 
in the fourth and fifth years of follow-up in schools for which data are 
available. 

Talent Development produced few statistically significant impacts on math achieve- 
ment. Figure 9 shows that Talent Development schools did not begin to improve their baseline 
average math total NCE scores until the fourth year of implementation. Fewer schools are in- 
cluded in the fourth year of analysis, but the improvement does not seem to be the result of 
dropping lower-performing schools. Instead, Talent Development schools seem to have made 
more progress than their comparison schools in the fourth and fifth years of implementation. 
This is particularly true in math problem solving, — which is expected, since the mathematics 
program recommended by Talent Development blends skill building with problem solving. Ta- 
ble 5 shows that, like the eighth-grade findings, the impacts for seventh grade are more pro- 
nounced in the reduction of the percentage of students scoring in the bottom quartile. Impacts 
on math problem solving were about 1 1 percentage points in Years 4 and 5 and were not driven 
by one or two schools. The five later-implementing schools showed a modestly positive start in 
math achievement in the first year of implementation. 
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The Talent Development Evaluation 
Table 5 

Impacts on SAT-9 Test Scores and Attendance 
for Seventh-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 



Impact at Follow-Up 8 Impact Effect Size b 



Outcome 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


SAT-9 test scores 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 


Math total 


Average NCE 


-0.6 


-0.5 


-1.6 


2.0 


0.3 


-0.04 


-0.04 


-0.11 


0.14 


0.02 


In the bottom quartile (%) 


-1.3 


-0.6 


3.1 


-10.1 * 


-5.3 


-0.03 


-0.01 


0.06 


-0.21 * 


-0.11 


At or above grade level (%) 


-0.3 


-2.6 


-3.9 


1.1 


-4.5 


-0.01 


-0.08 


-0.12 


0.03 


-0.14 


Math problem solving 


Average NCE 


0.0 


-0.5 


-1.5 


3.1 * 


0.8 


0.00 


-0.03 


-0.10 


0.20 * 


0.05 


In the bottom quartile (%) 


-1.1 


1.3 


4.0 


-H I ** 


-10.5 


-0.02 


0.03 


0.08 


-0.23 ** 


-0.22 


At or above grade level (%) 


0.8 


-1.7 


-4.3 


1.1 


-2.4 


0.03 


-0.05 


-0.14 


0.03 


-0.08 


Reading 


Average NCE 


0.0 


-0.5 


1.0 


1.6 


0.1 


0.00 


-0.03 


0.06 


0.09 


0.01 


In the bottom quartile (%) 


-0.4 


2.0 


-0.7 


-4.6 


-0.5 


-0.01 


0.04 


-0.01 


-0.09 


-0.01 


At or above grade level (%) 


2.2 


-1.3 


0.4 


3.2 


2.9 


0.06 


-0.03 


0.01 


0.08 


0.07 


Attendance' 


Number of school clusters 


5 


5 


5 


4 


2 


5 


5 


5 


4 


2 


Attendance rate 


-1.3 


0.0 


0.1 


K> 

oo 

* 

* 


0.3 


-0.04 


0.00 


0.00 


-0.09 ** 


0.01 


Attendance rate of 90% or higher (%) 


-3.0 


2.1 


5.7 * 


-1.6 


4.9 


-0.06 


0.04 


0.12 * 


-0.03 


0.10 


Attendance rate of 80% or lower (%) 


1.5 


-2.8 


-0.5 


2.7 


-2.3 


0.03 


-0.06 


-0.01 


0.06 


-0.05 



(continued) 
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Table 5 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

The number of school clusters varies by year due to the staggered implementation schedule and availability of data. SAT-9 test score results for Year 1 and Year 2 are 
pooled over six Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 3 are pooled over five Talent 
Development schools and five clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent 
Development comparison schools. 

Attendance measures were only available for five clusters. Therefore, results for Year 1, Year 2, and Year 3 are pooled over five Talent Development schools and five 
clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four clusters of non-Talent 
Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent Development comparison 
schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had 
repeated a prior grade. 

'‘The impact at follow-up for a given year was calculated as the difference in deviations from the baseline average between early-implementing Talent Development 
schools and their non-Talent Development comparison schools. A two-tailed t-test was applied to the impact at follow-up. Standard errors were adjusted to account for 
cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends in part on sample 
size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with Year 4 and Year 5, which include fewer schools. 

b The impact effect size for each outcome was calculated by dividing the impact in a given year by the standard deviation of that outcome for all 7th-grade students in 
the 11 Talent Development schools and 18 non-Talent Development comparison schools from a pre-Talent Development period, school years 1995-1996 and 1996- 
1997. 

'Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a 
given school year. Attendance measures were not available for one Talent Development school. School E. 
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Figure 9 

Impacts on SAT-9 Math Total NCE Scores 
for Seventh-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students’ school records front a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. 
Therefore, results for Year 1 and Year 2 are pooled over six Talent Development schools and six clusters of non-Talent 
Development comparison schools; results for Year 3 are pooled over five Talent Development schools and five clusters of 
non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development 
schools and two clusters of non-Talent Development comparison schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in SAT-9 math total NCE points from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development schools 
and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends 
in part on sample size, may be achieved with impacts of a smaller magnitude in the first two follow-up years, as compared 
with Year 4 and Year 5, which include fewer schools. 



- 51 - 





• Both Talent Development and non-Talent Development schools show 
modest improvements in seventh-grade reading achievement in later 
implementation years. Overall, there were no systematic differences be- 
tween these sets of schools over time. 

The third set of outcomes in Table 5 summarizes the impact of Talent Development on 
seventh-grade reading achievement. In the follow-up period, both Talent Development and non- 
Talent Development schools had very small deviations from the baseline average in reading, 
resulting in virtually no impact for Talent Development. Figure 10 shows some upswing in 
achievement in the last three years of implementation in Talent Development schools. However, 
on average, non-Talent Development comparison schools made similar gains in Year 5 that di- 
minish the impact of Talent Development. 

First-year findings for the five later-implementing schools show a stronger start in read- 
ing than in the early-implementing schools. Though impacts for this group are not statistically 
significant, the impact on average reading score is an increase of about 1 NCE. The analysis 
also shows an impact of 4 percentage points in the reduction of the percentage of students scor- 
ing in the bottom quartile and an impact of 4 percentage points in the increase of the percentage 
of students reading at grade level. 

• The findings indicate that Talent Development did not produce consis- 
tent impacts on seventh-grade attendance outcomes. 

Figure 1 1 shows attendance-rate deviations from baseline average for Talent Develop- 
ment and non-Talent Development schools. This figure illustrates the unstable pattern of im- 
pacts for seventh-grade attendance outcomes. The statistically significant and negative impact in 
Year 4 is driven by two Talent Development schools and is somewhat less reliable than the rela- 
tively flat pattern of impacts for the first three years of follow-up. There is some variability 
across schools in the size and direction of deviations from their baseline averages, but most 
changes are small, and no school appears to exhibit a pattern of improvement or decline in at- 
tendance rates. Similarly, there is little deviation from baseline average attendance rates for the 
five later-implementing Talent Development schools and their non-Talent Development com- 
parison schools. 



- 52 - 




The Talent Development Evaluation 
Figure 10 

Impacts on SAT-9 Reading NCE Scores 

for Seventh-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students front 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. 
Therefore, results for Year 1 and Year 2 are pooled over six Talent Development schools and six clusters of non-Talent 
Development comparison schools; results for Year 3 are pooled over five Talent Development schools and five clusters of 
non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development 
schools and two clusters of non-Talent Development comparison schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in SAT-9 reading NCE points from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development 
schools and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends 
in part on sample size, may be achieved with impacts of a smaller magnitude in the first two follow-up years, as compared 
with Year 4 and Year 5, which include fewer schools. 
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The Talent Development Evaluation 

Figure 11 

Impacts on Attendance Rates 

for Seventh-Grade Students in Early-Implementing Talent Development Schools, 

Five-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. Attendance data were not available 
for one Talent Development school, School F. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. 
Therefore, results for Year 1, Year 2, and Year 3 are pooled over five Talent Development schools and five clusters of non- 
Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four 
clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development 
schools and two clusters of non-Talent Development comparison schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in attendance rate from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development schools 
and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. Statistical significance, which depends 
in part on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared 
with Year 4 and Year 5. which include fewer schools. 
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Sizing Up the Results 

Taken together, what do these findings suggest about the Talent Development Middle 
School model? Two important patterns emerge: (1) The most prominent impacts occurred in 
mathematics achievement among eighth-grade students; and (2) the strength of impacts seems 
to be associated with the timing and intensity of Talent Development’s implementation. 

The most significant impacts were found for math achievement among eighth-grade 
students and were particularly strong in the later years of implementation. This pattern may re- 
flect a combination of factors: Eighth-graders in these later years may have benefited from cu- 
mulative years of exposure to Talent Development, and implementation may need two or more 
years to gain enough traction to produce significant results. Also, the development of math im- 
pacts before impacts in reading seems to be consistent with the nature of the curricular materials 
and teacher training in math, which focused on grade-specific content units and were readily 
transferable to classroom practice. 

The presence of Talent Development impacts seems to correspond with whether and 
when key components of the model were implemented. The components of the model were 
phased in over three years in the six schools that are the focus of this report. The impact analysis 
shows that improvements in student achievement, at least in math, began to emerge in Years 3 
and 4 of implementation. The math program was usually one of the first components imple- 
mented, and the professional development program is considered to be a two-year course of 
study by the model’s developers. Small learning communities were part of the first year of im- 
plementation in all schools; however, impacts on attendance do not seem to follow the imple- 
mentation of this component, which was expected to increase student engagement. 

The relationship between implementation and effects is consistent with findings from a 
meta-analysis of the 29 most widely discussed and disseminated models of comprehensive 
school reform (CSR). The CSR study found that substantial improvements in student achieve- 
ment typically do not emerge, if they occur at all, until several years after the reforms are initi- 
ated. CSR effects began to increase substantially after the fifth year of implementation. 38 The 
same study suggests that the magnitude of the impacts reported here are comparable with other 
third-party evaluations of CSR models that have strong evidence of effectiveness. 

Impact estimates on average math achievement for eighth-grade students range between 
1 NCE and 4 NCEs, corresponding to effect sizes of between 0.04 and 0.28. Although no abso- 
lute standard exists to define whether a specific effect size is large or small, there are some tradi- 
tional guidelines. This report’s effect sizes of the impacts for eighth-grade math achievement 
fall in the small-to-moderate categories. More recent analyses suggest that this categorization 

3S Borman, Hewes, Overman, and Brown, 2003, p. 152. 
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may be too conservative for educational outcomes. It is possible for small-to-moderate effect 
sizes to have substantial education significance. For example, if all 38 nonselective middle 
schools in the district exacted the most promising impacts on math achievement described in 
this report, more than 1,200 eighth-grade students could move out of the bottom quartile in 
math achievement each year. 

Although the early impact findings in this report should be considered preliminary — 
because this study focuses on six middle schools in a single district with three to five years of 
follow-up data — they are encouraging, particularly for math achievement among eighth-grade 
students. It is not common to find early impacts of this magnitude in evaluations of models of 
comprehensive school reform. 39 



Next Steps 

Overall, the findings in this report suggest that the Talent Development Middle School 
model has positive and significant impacts on certain measures, particularly when key compo- 
nents have been adequately implemented. And this may offer promise that the model will have 
positive and significant impacts on other outcomes in the future, but more data collection and 
analysis need to be done. 

A subsequent report from MDRC on the Talent Development Middle School model 
will track outcomes for two more years of implementation in the six early-implementing 
schools and in the five later-implementing schools in an effort to answer remaining questions, 
including: 

• Will the improvements in eighth-grade mathematics be strengthened over 
time? Will the impacts continue to be sustained in early-implementing 
schools, and will those impacts eventually accrue in later-implementing 
schools and for students in other grade levels? 

• Will the preliminary findings that show some promise for Talent Develop- 
ment’s impact on reading achievement be realized? 

• Do the benefits of Talent Development accrue primarily to those students 
who regularly attend school in both the baseline and the follow-up periods? 

(If so, this would explain the seemingly contradictory findings about atten- 
dance and achievement.) 



39 Borman, Hewes, Overman, and Brown, 2003; Bloom, 2001. 
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• Will improvements in achievement during middle school years translate into 
students’ greater persistence in high school and their eventual graduation? 

• How resilient is the Talent Development model in the face of significant 
changes in the district’s context, including management of some schools by 
outside organizations? 

The upcoming report, due in 2005, will be produced in the context of a range of com- 
prehensive school reform research sponsored by the U.S. Department of Education. Taken to- 
gether, this research has the potential to deliver a powerful message to policymakers, research- 
ers, and practitioners about what interventions help to improve student performance and atten- 
dance in low-performing secondary schools. 
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The Talent Development Evaluation 
Appendix Table A.l 

Year-by-Year Levels and Impacts on SSA Math Test Score Outcomes 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 




Year 4 


Year 5 


Year 1 


Year 2 




Year 3 


Year 4 


Year 5 


Average NCE 


26.1 


27.3 


29.2 




31.9 


34.1 


25.5 


27.4 




27.8 


28.6 


30.4 


Deviation from baseline 


2.6 ** 


3 g *** 


5.7 




g 2 *** 


7 7 *** 


1.3 *** 


3.2 




3.6 *** 


4.3 *** 


3.6 *** 


At or above grade level (%) 


5.4 


5.8 


7.3 




11.4 


13.7 


5.9 


7.0 




6.3 


6.2 


8.0 


Deviation from baseline 


0.8 


1.2 


2.7 


** 


6.2 *** 


6.2 ** 


1.4 ** 


2.4 


*** 


1.8 *** 


1.7 * 


1.7 


In the bottom quartile (%) 


77.9 


72.1 


72.3 




64.4 


57.5 


77.8 


74.0 




73.8 


73.4 


66.8 


Deviation from baseline 


-5.4 * 


-11.2 *** 


-11.0 




-19.5 *** 


-20 2 *** 


_2 7 ** 


-6.5 


*** 


-6.7 *** 


_7 4 *** 


_g 3 *** 


Number of school clusters 


6 


6 


6 




4 


2 


6 


6 




6 


4 


2 





II. Impact of Talent Development Compared with 
Non-Talent Development Schools 




III. Impact Effect Size 






Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


1.3 


0.6 


2.2 * 


3 9 *** 


4.1 * 


0.09 


0.04 


0.15 * 


0 27 *** 


0.28 * 


At or above grade level (%) 


-0.6 


-1.2 


0.9 


4.5 ** 


4.5 


-0.03 


-0.06 


0.04 


0.20 ** 


0.20 


In the bottom quartile (%) 


- 2.8 


-A.l 


-4.4 


-12.1 *** 


-10.9 


-0.07 


-0.12 


-0.11 


-0.30 *** 


-0.28 


Number of school clusters 


6 


6 


6 


4 


2 


6 


6 


6 


4 


2 



(continued) 
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Appendix Table A.l (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average deviation 
of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1, Year 2, and Year 3 are pooled 
over six Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools 
and four clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent 
Development comparison schools. 

Panel 11: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was calculated as 
the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel 111: The single row of data from Panel 11 continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was calculated by 
dividing the impact at follow-up by the standard deviation of the outcome for all 8th-grade students in the 18 non-Talent Development comparison schools from school years 
1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with 
Year 4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 6 schools, 
while average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 
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The Talent Development Evaluation 
Appendix Table A.2 

Year-by-Year Levels and Impacts on SSA Reading Test Score Outcomes 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


27.8 


30.6 


29.3 


30.8 


36.2 


28.8 


27.8 


29.0 


30.1 


32.7 


Deviation from baseline 


-0.6 


2.1 ** 


0.8 


2.1 


3.7 ** 


0.5 


-0.5 


0.7 


14 ** 


0.9 


At or above grade level (%) 


8.7 


11.4 


8.7 


9.9 


14.7 


8.7 


7.4 


8.2 


8.6 


10.2 


Deviation from baseline 


-0.2 


2.5 


-0.2 


0.7 


1.7 


-0.3 


-1.5 * 


-0.7 


-1.1 


-2.4 


In the bottom quartile (%) 


73.3 


68.3 


71.8 


66.0 


52.6 


70.2 


74.0 


71.4 


68.7 


62.1 


Deviation from baseline 


3.0 


-2.0 


1.5 


-3.8 


-9.3 * 


-0.2 


3.6 *** 


1.0 


-0.5 


-0.6 


Number of school clusters 


6 


6 


6 


4 


2 


6 


6 


6 


4 


2 




II. Impact of Talent Development Compared with 
Non-Talent Development Schools 




III. Impact Effect Size 






Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


-1.1 


2.7 ** 


0.1 


0.7 


2.8 


-0.07 


0.17 ** 


0.01 


0.05 


0.17 


At or above grade level (%) 


0.0 


4.0 ** 


0.6 


1.8 


4.0 


0.00 


0.14 ** 


0.02 


0.06 


0.14 


In the bottom quartile (%) 


3.2 


-5.6 * 


0.5 


-3.3 


-8.8 


0.07 


-0.12 * 


0.01 


-0.07 


-0.19 


Number of school clusters 


6 


6 


6 


4 


2 


6 


6 


6 


4 


2 



(continued) 
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Appendix Table A.2 (continued) 

SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel 1: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average deviation 
of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1, Year 2, and Year 3 are pooled 
over six Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools 
and four clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent 
Development comparison schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was calculated as 
the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel III: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was calculated by 
dividing the impact at follow-up by the standard deviation of the outcome for all 8th-grade students in the 18 non-Talent Development comparison schools from school years 
1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with 
Year 4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 6 schools, 
while average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 




The Talent Development Evaluation 
Appendix Table A.3 

Year-by-Year Levels and Impacts on Attendance Rate Outcomes 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline Average 
Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 




Year 3 




Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Attendance rate 


86.9 


87.6 




87.4 




85.5 


87.7 


85.5 


85.8 


85.3 


85.7 


88.1 


Deviation from baseline 


3 o *** 


3.8 


*** 


3.5 


*** 


1.9 


2.0 


14 ** 


j 7 *** 


1.2 ** 


2 o *** 


3 Q *** 


Attendance rate of 90% or higher (%) 


55.1 


54.5 




57.8 




52.9 


61.4 


49.8 


51.1 


49.1 


51.0 


57.9 


Deviation from baseline 


8.3 ** 


7.6 


** 


10.9 




8.0 * 


10.4 


2.0 


3.2 * 


1.2 


3.9 * 


OO 

* 

* 


Attendance rate of 80% or lower (%) 


21.8 


20.8 




20.5 




24.0 


17.7 


25.0 


24.2 


25.6 


25.0 


20.6 


Deviation from baseline 


-6.7 ** 


-7.7 


*** 


- 8.0 




-5.2 * 


-3.7 


2 9 ** 


_3 g *** 


-2.3 * 


-3.9 ** 


- 6.2 *** 


Number of school clusters 


5 


5 




5 




4 


2 


5 


5 


5 


4 


2 





II. Impact of Talent Development Compared with 
Non-Talent Development Schools 

Year 1 Year 2 Year 3 Year 4 Year 5 


Year 1 


III. Impact Effect Size 
Year 2 Year 3 Year 4 


Year 5 


Attendance rate 


1.6 


2.1 * 


2.3 * 


- 0.1 


- 1.0 


0.05 


0.07 * 


0.08 * 


0.00 


-0.03 


Attendance rate of 90% or higher (%) 


6.3 


4.4 


9 7 ** 


4.0 


2.0 


0.13 


0.09 


0.20 ** 


0.08 


0.04 


Attendance rate of 80% or lower (%) 


-3.8 


-3.9 


-5.7 * 


-1.3 


2.5 


-0.08 


-0.08 


- 0.12 * 


-0.03 


0.05 


Number of school clusters 


5 


5 


5 


4 


2 


5 


5 


5 


4 


2 
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Appendix Table A.3 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 
Attendance data were not available for the sixth early-implementing Talent Development school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average 
deviation of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1, Year 2, and Year 3 are 
pooled over five Talent Development schools and five clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent 
Development schools and four clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two 
clusters of non-Talent Development comparison schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel III: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was 
calculated by dividing the impact at follow-up by the standard deviation of the outcome for all 8th-grade students in the 18 non-Talent Development comparison schools 
from school years 1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared 
with Year 4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 
5 schools, while average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a 
given school year. Attendance data were unavailable for one Talent Development school. School F. 
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The Talent Development Evaluation 
Appendix Table A.4 

Year-by-Year Levels and Impacts on One- Year Promotion Outcomes 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline Average 
Talent Development Schools Non-Talent Development Schools 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Promoted to 9th grade (%) 


97.7 


98.0 


100.0 


98.3 


97.6 


98.2 


97.3 


97.6 


96.7 


97.9 


Deviation from baseline 


0.4 


0.7 


2 2 ** 


0.9 


0.2 


0.3 


-0.5 


-0.2 


-1.2 


0.1 


Number of school clusters 


6 


6 


6 


4 


2 


6 


6 


6 


4 


2 





II. Impact of Talent Development Compared with 
Non-Talent Development Schools 

Year 1 Year 2 Year 3 Year 4 Year 5 


Year 1 


III. Impact Effect Size 
Year 2 Year 3 Year 4 


Year 5 


Promoted to 9th grade (%) 
Number of school clusters 


0.0 

6 


1.1 

6 


2.5 ** 2.1 

6 4 


0.1 

2 


0.00 

6 


0.05 0.11 ** 0.09 

6 6 4 


0.01 

2 



(continued) 
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Appendix Table A.4 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average 
deviation of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1, Year 2, and Year 3 are 
pooled over six Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent 
Development schools and four clusters of non-Talent Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two 
clusters of non-Talent Development comparison schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel III: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was 
calculated by dividing the impact at follow-up by the standard deviation of the outcome for all 8th-grade students in the 18 non-Talent Development comparison schools 
from school years 1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I. and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with 
Year 4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 6 
schools, while average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 

Eighth-grade students were considered promoted if they were listed as 9th-grade students in the district's administrative data file one year after the current year. 
Students whose records were not included in the data file one year after the current year, for whatever reason, were not in the analysis for this outcome. 
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The Talent Development Evaluation 
Appendix Table B.l 

Year-by-Year Levels and Impacts on SAT-9 Math Test Score Outcomes 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 











I. Outcome Levels Compared with Baseline 












Talent Development Schools 






Non-Talent Development Schools 




Outcome 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


34.4 


33.7 


33.7 


36.0 


39.5 


34.4 


33.6 


34.5 


33.9 


38.1 


Deviation from baseline 


0.4 


-0.3 


- 0.6 


2.9 * 


3.8 


1.1 * 


0.2 


1.0 


0.9 


3.5 *** 


At or above grade level (%) 


16.0 


12.1 


11.8 


13.3 


19.4 


14.7 


13.0 


13.7 


10.6 


19.9 


Deviation from baseline 


1.9 


- 2.0 


-2.9 


- 0.2 


0.5 


2.2 ** 


0.5 


1.0 


-1.3 


5.0 ** 


In the bottom quartile (%) 


57.4 


61.5 


62.3 


54.8 


44.5 


59.0 


62.5 


59.8 


63.1 


49.0 


Deviation from baseline 


-4.3 


- 0.2 


1.3 


_9 7 ** 


-13.2 * 


-3.1 * 


0.5 


- 1.8 


0.4 


_7 9 ** 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 




II. Impact of Talent Development Compared with 
Non-Talent Development Schools 




III. Impact Effect Size 






Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


- 0.6 


-0.5 


- 1.6 


2.0 


0.3 


-0.04 


-0.04 


- 0.11 


0.14 


0.02 


At or above grade level (%) 


-0.3 


- 2.6 


-3.9 


1.1 


-4.5 


- 0.01 


-0.08 


- 0.12 


0.03 


-0.14 


In the bottom quartile (%) 


-1.3 


- 0.6 


3.1 


- 10.1 * 


-5.3 


-0.03 


- 0.01 


0.06 


- 0.21 * 


- 0.11 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 
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Appendix Table B.l (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average deviation 
of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1 and Year 2 are pooled over six 
Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 3 are pooled over five Talent Development schools and five 
clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four clusters of non-Talent Development 
comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent Development comparison schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was calculated 
as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel III: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was calculated 
by dividing the impact at follow-up by the standard deviation of the outcome for all 7th-grade students in the 18 non-Talent Development comparison schools from school 
years 1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with 
Year 4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 6 
schools, while average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 
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The Talent Development Evaluation 
Appendix Table B.2 

Year-by-Year Levels and Impacts on SAT-9 Math Problem Solving Test Score Outcomes 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


35.4 


34.4 


34.8 


37.3 


41.3 


34.4 


34.0 


35.2 


33.8 


38.6 


Deviation from baseline 


1.4 


0.4 


0.6 


4 2 *** 


5.6 ** 


1.4 ** 


0.9 * 


2 j *** 


1.2 


4 g *** 


At or above grade level (%) 


17.3 


14.1 


13.8 


14.9 


23.5 


14.3 


13.6 


15.7 


11.8 


21.8 


Deviation from baseline 


2.8 


-0.4 


- 1.1 


1.4 


5.7 


2.0 * 


1.3 


2 2 *** 


0.3 


8.1 *** 


In the bottom quartile (%) 


56.2 


59.8 


58.6 


49.7 


34.8 


59.3 


60.4 


56.5 


60.4 


48.1 


Deviation from baseline 


-5.0 


-1.4 


-2.4 


-14.7 *** 


-22 9 *** 


-3.9 ** 


- 2.8 * 


4 *** 


-3.6 


-12.4 *** 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 




II. Impact of Talent Development Compared with 
Non-Talent Development Schools 




III. 


Impact Effect Size 






Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


0.0 


-0.5 


-1.5 


3.1 * 


0.8 


0.00 


-0.03 


- 0.10 


0.20 * 


0.05 


At or above grade level (%) 


0.8 


-1.7 


-4.3 


1.1 


-2.4 


0.03 


-0.05 


-0.14 


0.03 


-0.08 


In the bottom quartile (%) 


- 1.1 


1.3 


4.0 


- 11.1 ** 


-10.5 


- 0.02 


0.03 


0.08 


-0.23 ** 


- 0.22 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 



(continued) 
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Appendix Table B.2 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average deviation 
of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1 and Year 2 are pooled over six 
Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 3 are pooled over five Talent Development schools and five 
clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four clusters of non-Talent Development 
comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent Development comparison schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was calculated as 
the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel III: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was calculated by 
dividing the impact at follow-up by the standard deviation of the outcome for all 7th-grade students in the 18 non-Talent Development comparison schools from school years 
1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with Year 
4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 6 schools, while 
average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 
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The Talent Development Evaluation 
Appendix Table B.3 

Year-by-Year Levels and Impacts on SAT-9 Reading Test Score Outcomes 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


36.2 


35.2 


37.6 


37.5 


42.3 


36.2 


35.7 


36.2 


35.9 


41.5 


Deviation from baseline 


0.1 


- 0.8 


1.0 


1.5 


3.5 


0.2 


-0.3 


- 0.1 


- 0.1 


3 3 *** 


At or above grade level (%) 


22.2 


19.1 


23.0 


21.6 


30.7 


20.0 


20.4 


22.1 


19.2 


27.6 


Deviation from baseline 


0.4 


-2.7 


0.3 


0.4 


4.0 


- 1.8 * 


-1.4 


- 0.2 


- 2.8 * 


1.2 


In the bottom quartile (%) 


50.5 


54.5 


49.3 


47.3 


37.6 


50.6 


52.3 


51.4 


52.2 


39.8 


Deviation from baseline 


-1.3 


2.7 


-0.4 


-4.1 


- 6.6 


-0.9 


0.7 


0.3 


0.5 


- 6.1 ** 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 




II. Impact of Talent Development Compared with 
Non-Talent Development Schools 




III. 


Impact Effect Size 






Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Average NCE 


0.0 


-0.5 


1.0 


1.6 


0.1 


0.00 


-0.03 


0.06 


0.09 


0.01 


At or above grade level (%) 


2.2 


-1.3 


0.4 


3.2 


2.9 


0.06 


-0.03 


0.01 


0.08 


0.07 


In the bottom quartile (%) 


-0.4 


2.0 


-0.7 


-4.6 


-0.5 


- 0.01 


0.04 


- 0.01 


-0.09 


- 0.01 


Number of school clusters 


6 


6 


5 


4 


2 


6 


6 


5 


4 


2 



(continued) 




- 75 - 



Appendix Table B.3 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average deviation of 
that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1 and Year 2 are pooled over six 
Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 3 are pooled over five Talent Development schools and five 
clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four clusters of non-Talent Development 
comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent Development comparison schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was calculated as 
the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel III: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was calculated by 
dividing the impact at follow-up by the standard deviation of the outcome for all 7th-grade students in the 18 non-Talent Development comparison schools from school years 
1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated as: 

*** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with Year 
4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 6 schools, while 
average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 
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Appendix Table B.4 

Year-by-Year Levels and Impacts on Attendance Rate Outcomes 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results 



Outcome 




I. Outcome Levels Compared with Baseline 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Attendance rate 


85.5 


87.0 


86.4 


84.4 


88.1 


85.8 


86.0 


85.3 


86.2 


86.9 


Deviation from baseline 


-0.7 


0.8 


0.3 


-1.5 


0.1 


0.7 


0.8 ** 


0.2 


1.2 ** 


-0.2 


Attendance rate of 90% or higher (%) 


46.9 


51.5 


52.7 


47.9 


56.4 


49.4 


48.8 


46.4 


50.5 


52.1 


Deviation from baseline 


-2.9 


1.6 


2.9 


0.0 


1.7 


0.1 


-0.5 


2 9 ** 


1.6 


-3.2 * 


Attendance rate of 80% or lower (%) 


24.7 


21.5 


24.5 


26.7 


19.6 


25.2 


26.3 


27.0 


26.0 


24.4 


Deviation from baseline 


0.7 


-2.5 


0.5 


2.2 


0.2 


-0.9 


0.2 


0.9 


-0.5 


2.5 * 


Number of school clusters 


5 


5 


5 


4 


2 


5 


5 


5 


4 


2 




II. Impact of Talent Development Compared with 
Non-Talent Development Schools 




III. Impact Effect Size 






Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Attendance rate 


-1.3 


0.0 


0.1 


-2.8 ** 


0.3 


-0.04 


0.00 


0.00 


-0.09 ** 


0.01 


Attendance rate of 90% or higher (%) 


-3.0 


2.1 


5.7 * 


-1.6 


4.9 


-0.06 


0.04 


0.12 * 


-0.03 


0.10 


Attendance rate of 80% or lower (%) 


1.5 


-2.8 


-0.5 


2.7 


-2.3 


0.03 


-0.06 


-0.01 


0.06 


-0.05 


Number of school clusters 


5 


5 


5 


4 


2 


5 


5 


5 


4 


2 



(continued) 
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Appendix Table B.4 (continued) 

SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Panel 1: Each outcome has two rows of data. The first row shows the average level for that outcome in each follow-up year. The second row shows the average 
deviation of that outcome from the three-year pre-implementation baseline average in each follow-up year. 

The number of school clusters varies by year due to the staggered implementation schedule and the availability of data. Results for Year 1 and Year 2 are pooled over 
six Talent Development schools and six clusters of non-Talent Development comparison schools; results for Year 3 are pooled over five Talent Development schools and 
five clusters of non-Talent Development comparison schools; results for Year 4 are pooled over four Talent Development schools and four clusters of non-Talent 
Development comparison schools; and results for Year 5 are pooled over two Talent Development schools and two clusters of non-Talent Development comparison 
schools. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each follow-up year. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Panel 111: The single row of data from Panel II continues into Panel III to show the effect sizes for the impact estimates of Panel II. The impact effect size was 
calculated by dividing the impact at follow-up by the standard deviation of the outcome for all 7th-grade students in the 18 non-Talent Development comparison schools 
from school years 1995-1996 and 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether the student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, and to the 
impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance levels are indicated 
as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Statistical significance, which in part depends on sample size, may be achieved with impacts of a smaller magnitude in the first three follow-up years, as compared with 
Year 4 and Year 5, which include fewer schools. Similarly, note that average deviations from baseline for Talent Development schools (in Panel I) include at most 5 
schools, while average deviations from baseline for non-Talent Development schools (also in Panel I) include up to 18 schools. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a 
given school year. Attendance data were unavailable for one Talent Development school, School F. 
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MDRC is a nonprofit, nonpartisan social policy research organization. We are dedicated 
to learning what works to improve the well-being of low-income people. Through our 
research and the active communication of our findings, we seek to enhance the 
effectiveness of social policies and programs. MDRC was founded in 1974 and is located 
in New York City and Oakland, California. 

MDRC’s current projects focus on welfare and economic security, education, and 
employment and community initiatives. Complementing our evaluations of a wide range 
of welfare refonns are new studies of supports for the working poor and emerging 
analyses of how programs affect children’s development and their families’ well-being. 

In the field of education, we are testing reforms aimed at improving the performance of 
public schools, especially in urban areas. Finally, our community projects are using 
innovative approaches to increase employment in low-income neighborhoods. 

Our projects are a mix of demonstrations — field tests of promising program models — 
and evaluations of government and community initiatives, and we employ a wide range 
of methods to determine a program’s effects, including large-scale studies, surveys, case 
studies, and ethnographies of individuals and families. We share the findings and lessons 
from our work — including best practices for program operators — with a broad 
audience within the policy and practitioner community, as well as the general public and 
the media. 

Over the past quarter century, MDRC has worked in almost every state, all of the nation’s 
largest cities, and Canada. We conduct our projects in partnership with state and local 
governments, the federal government, public school systems, community organizations, 
and numerous private philanthropies. 
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Unit I 

Analytic Appendix 




Introduction 

This appendix outlines the analytic approach used by MDRC to estimate the Talent De- 
velopment Middle School model’s impact on student performance and attendance. Impact find- 
ings are from Talent Development’s first five years of implementation in a large, urban school 
district. The full report — which discusses the findings in detail and describes the components 
of Talent Development and the context in which it operates — is available online at this Web 
site or from MDRC as a printed document. 

Talent Development is a comprehensive reform model for large middle schools that 
serve high-poverty populations and face serious problems with student attendance, discipline, 
and achievement scores. The model calls for specific changes in school organization and curric- 
ula with the goals of establishing a strong, positive school climate for learning; promoting high 
standards for mathematics, language arts, science, and U.S. history coursework for all students; 
and providing professional development systems to support implementation of the recom- 
mended reforms. Each of these changes is aimed specifically at enhancing student attendance in 
school, improving measurable student learning, and keeping students on course toward grade- 
level promotion and a successful transition to high school. 

The impact analysis for this report focuses on engagement and performance outcomes 
for seventh- and eighth-grade students. The three types of outcomes that are examined are the 
ones likely to be in greatest proximity to the early phases of Talent Development’s implementa- 
tion: daily attendance, reading and math test scores, and grade-level promotion. The analytic 
approach used to measure the model’s impact on these outcomes can best be described as a 
comparative interrupted time series design. 1 

Before detailing the steps of the design, it is important to distinguish between this 
study’s measures of program outcomes and its measures of program impacts. The term “out- 
comes” here refers to the status or behavior of individual students or groups of students at vari- 
ous points during the period under study. In this study, the outcomes are measures of student 
attendance, grade-level promotion, and test scores. The term “impact” here refers to Talent De- 
velopment’ s effect on an outcome. 

Constructing a Counterfactual 

In this study, the average outcome levels (or even year-to-year changes in outcomes) for 
students in the Talent Development schools, by themselves, provide potentially misleading in- 



*For further discussion of using interrupted time series analysis to measure impacts of whole-school re- 
form, see also Bloom (2003) and Snipes (2003). 
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dications of Talent Development’s impacts. Previous research has shown that students within a 
school or set of schools may improve from year to year or may differ from other students for 
reasons not necessarily related to a special intervention like Talent Development. The ideal re- 
search situation would allow for an absolutely reliable estimate of the student performance lev- 
els that would have been observed in the absence of the intervention, that is, the counte if actual, 
and would compare this with actual student performance. Random assignment is the most reli- 
able basis from which to construct estimates of the counterfactual. However, since random as- 
signment was not possible for this evaluation, the comparative interrupted time series analysis 
attempted to construct the best counterfactual possible short of random assignment, in order to 
estimate the true impact of Talent Development. 

To this end, it was necessary to compare the experiences of a group of students who 
were exposed to Talent Development against the experiences of a similar group of students who 
were not. The more comparable the two groups are prior to the introduction of Talent Develop- 
ment, the more likely it is that later differences can be attributed to the program. Moreover, us- 
ing this kind of comparison makes it possible to account for factors other than Talent Develop- 
ment that may have caused a change or difference in student engagement and performance. 

The Logic of the Comparative Interrupted Time Series Design 

The comparative interrupted times series design consists of an interrupted time series 
analysis and a comparison school analysis, each of which builds on the strengths of the other 
and addresses each other' s potential limitations. Together, the two parts of the design construct 
a counterfactual for the evaluation. Specifically, the interrupted time series assesses the extent to 
which measures of engagement and performance for students in Talent Development schools 
differ from measures of engagement and performance for similar students in the same schools 
prior to Talent Development implementation. The analysis of comparison schools looks at Tal- 
ent Development schools versus non-Talent Development schools (which are similar middle 
schools in the same district that are not implementing the reform model). 

The first analysis provides an indication of whether the participating middle schools ex- 
perience a deviation from their historical patterns in student outcomes coincident with the intro- 
duction of Talent Development. The projection of each middle school’s recent history acts as the 
counterfactual. This is a particularly good counterfactual because, in the absence of the reform, 
many aspects of the school would be expected to stay the same: students, faculty, polices, school 
culture, neighborhood, and physical plant. Using a historical pattern as a counterfactual has the 
potential to control for both measurable and unmeasurable characteristics of a given school. 

However, the deviation from the baseline alone may not necessarily reflect the impact 
of Talent Development. Similar deviations from historical patterns could have been caused by 
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districtwide policies or interventions that occurred at about the same time as Talent Develop- 
ment implementation. For example, while Talent Development scaled up, the district that is the 
focus of this study participated in an NSF Urban Systemic Initiative for mathematics . 2 Such an 
effort may have caused positive deviations from baseline averages in math achievement at mid- 
dle schools in the district. An interrupted time series design would capture this improvement 
and ascribe it to Talent Development as an impact of the program. Talent Development may 
have caused some, all, or none of this change in math achievement. In order to sort out what 
part of the deviation from baseline is due to Talent Development, the analysis looks at similar 
middle schools in the same district. 

The second analysis in the comparative interrupted time series design (the comparison 
between Talent Development and non-Talent Development schools) helps to account for other 
factors in the broader school district that may influence school functioning and student engage- 
ment and performance. For this part of the analysis, Talent Development schools are matched 
with sets of comparison schools that are similar on several dimensions, including racial/ethnic 
composition and test scores. The Talent Development and comparison schools are all nonselec- 
tive, comprehensive middle schools in the same large, urban district. Measures of student 
achievement and engagement at the comparison schools provide a good indication of what 
might have been observed in Talent Development schools in the absence of the intervention. 

It should be noted, however, that differences between the Talent Development and 
comparison schools alone do not necessarily reflect the impact of Talent Development. Some 
differences could be an artifact of differences in the prior trends in student engagement and per- 
formance. For example, test scores for students in Talent Development schools may actually 
have been lower than those of students in non-Talent Development schools, and they might 
have improved only marginally after Talent Development began. At the same time, test scores 
for students attending similar schools in the district may actually have been declining over the 
same period. In such an instance, Talent Development would have a positive impact by prevent- 
ing test scores from dropping, rather than by improving the overall average. This could be ob- 
served only by comparing an interrupted time series for both Talent Development and non- 
Talent Development comparison schools. 

The comparative interrupted time series design makes this comparison by estimating 
the deviations from the historical patterns for the Talent Development schools and subtracting 
from these the deviations from historical patterns for similar non-Talent Development middle 
schools during the same period. The differences between these deviations constitute Talent De- 
velopment’s impact on student outcomes. When combined with regression analysis to control 



2 To preserve the anonymity of the district and schools included in this study, this appendix refers geneti- 
cally to “the district” and uses the labels “School A” through “School F.” 
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for differences due individual student background characteristics and prior school experiences, 
the approach isolates the unique impact that Talent Development has on student engagement 
and performance. 

It should be noted, however, that the comparative interrupted time series approach still 
has limitations that are present in all quasi-experimental designs. In this case, projection of a 
baseline average for a given school may not be a reliable predictor of future student outcomes. 
Also, finding comparison schools for the Talent Development schools is limited to observable 
characteristics of the student body and may miss important factors that affect student outcome 
trajectories. In addition, multiple -regression techniques control for compositional changes in 
measurable student characteristics, but there may changes in unmeasurable student characteris- 
tics that correlate with student outcomes. Finally, there still may be alternative explanations or 
other factors unrelated to Talent Development that contribute to the observed differences in stu- 
dent outcomes. For example, the analysis does not account for the process by which schools 
enter into the Talent Development network. Some may argue that schools with more entrepre- 
neurial leaders, who are more likely to seek out a reform model like Talent Development, may 
experience improved student outcomes even in the absence of the intervention. The analysis is 
unable to rule out this possibility. Despite the limitations of the comparative interrupted time 
series approach, it offers a valid estimate of the impact of Talent Development in middle 
schools in the district, particularly when estimates are pooled across several schools. 

The primary source of data for this analysis is individual students’ school records, 
which were obtained from the district. Table 1 provides a list of the types of data that were ob- 
tained for this evaluation and the school years and grade levels for which they are available. In 
general, administrative, attendance, and course-detail information is available for all middle 
school students in the district beginning with the 1995-1996 school year through the 2001-2002 
school year. 3 Table 2 describes the types of information included in these data sets. Table 3 de- 
fines several key outcomes included in the analysis. 

The rest of this appendix provides a step-by-step description of the analyses. The fol- 
lowing section details the steps that make up the interrupted time series approach, including es- 
timating deviations from baseline for Talent Development schools, controlling for composi- 
tional shifts, and accounting for cohort effects. The section after that describes the comparison- 
school approach, including selecting comparison schools and estimating their deviations from 
baseline. The final section describes estimating impacts and pooling estimates across schools. 



3 Student attendance records were not consistently available for one Talent Development middle school in 
the district. School F. Data from this school could not be included in estimates of Talent Development’s impact 
on attendance outcomes for seventh- and eighth-grade students. 
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The Talent Development Evaluation 

Analytic Appendix 
Table 1 

Data Sources and Availability, 
by School Year and Grade Level 



School Year 



Data Source 


1991-1992 


1992-1993 


1993-1994 


1994-1995 


1995-1996 


1996-1997 


1997-1998 


1998-1999 


1999-2000 


2000-2001 


2001-2002 


Administrative records 










9-12 


6-12 


6-12 


6-12 


6-12 


6-12 


6-12 


Attendance records 










9-12 


6-8 


6-12 


6-12 


6-12“ 


6-12“ 


6-12 


Course-detail records 










6-12 


6-12 


6-12 


6-12 


6-12 


6-12 


6-12 


Test scores 
CTBS 


1-8 


1-8 


1-8 


4-8 b 
















SAT-9 










2-4, 6-8, 11 


2-4, 6-8, 11 


2-4, 7-8 


2-4, 7-8 


2-4, 7-8, 
10-11 


3, 4, 7, 8 


3, 4, 7, 10 


SSA 










5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 


5, 8, 11 



SOURCE: Individual students' school records from a large, urban school district. 

NOTES: Blank spaces indicate that no records are available for those years. 

Administrative records include information on students' race; gender; birth date; and final school-enrollment status for the year, including withdrawal and dropout status and 
number of suspensions. 

Attendance records include information on the number of days a student is present and absent for each marking period. Unless otherwise noted, this sample includes students 
who attended at least one day in any of the marking periods. 

Course-detail records include information on credits attempted, credits earned, grades, and absences for each course in which a student was enrolled during the year. Unless 
otherwise noted, this sample includes students who were enrolled in at least one course during the year, according to the course-detail records. 

Test scores may not be available for every student. 

Comprehensive Test of Basic Skills (CTBS) records include test scores for reading, math, science, and social studies. 

Stanford Achievement Test, Ninth Edition (SAT-9) records include test scores for reading, math, problem solving, procedures, and science. 

State Standards Assessment (SSA) records include test scores for reading and math. 

“Attendance records for years 1999-2000 and 2000-2001 include only students who were present for at least one day in the last marking period. 

b Test scores for 1994-1995 are missing for a number of middle and high schools. 
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Analytic Appendix 
Table 2 



Data Types and Descriptions 



Data Type 


Description 


1 1 


Administrative 


Administrative data typically include student background information, such as birth date, race, 
and gender, as well as information on school enrollment status, special education classification, 
and English Speakers of Other Languages (ESOL) training. The administrative data are also the 
primary source of information about the grade level in which students were enrolled during each 
school year. These records were used to determine whether students were promoted from year to 
year or retained in grade. Typically, administrative records are available for all students in a 
given school level regardless of whether they entered the district after the school year began or 
whether they dropped out or left the district before the end of the year. 


1 1 


Attendance 


Attendance data include information about the number of days a student was present or absent 
during a given school year. In some years, these data were provided on a quarterly basis, and in 
other years they were provided as cumulative records. This information was used to construct an 
attendance rate and an absentee rate for each student in the files. Typically, the attendance files 
include only students who were present for at least one day during the final marking period of 
the year. This means that students who dropped out of school or who left the district before the 
start of the final marking period do not have an attendance record for this analysis. 


1 I 


Course-detail 

records 


Course-detail records include, for each course in which a student was enrolled during a given 
school year, the course code number, an abbreviated name, the number of credits the student 
attempted, the number of credits the student earned, and the grade the student received. For each 
student in the file, this information was used to construct both an annual and a cumulative count 
of credits earned and attempted. The information was also used to calculate credits earned in 
particular subject areas. 


1 1 


Test scores 
(nationally 
normed) 


The California Test of Basic Skills (CTBS) and the Stanford Achievement Test, Ninth Edition 
(SAT-9), are norm-referenced test scores, which provide information on individual student 
achievement relative to scores obtained from a random sample of students from across the 
country. SAT-9 scores in math and reading are available as Normal Curve Equivalents, National 
Percentiles, and Scale Scores. In general, these test scores were used in the analysis to control 
for student achievement prior to entering high school. 


1 1 


Test scores 
(state) 


The State Standards Assessment (SSA) is a criterion-referenced test, which provides information 
on student skills and content knowledge specified by the state. SSA test scores in math and 
reading are available as Normal Curve Equivalents (NCEs), State Percentiles, and Scale Scores 
for each of these grades and the school years listed in Analytic Appendix Table 1. 
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The Talent Development Evaluation 

Analytic Appendix 
Table 3 



Definitions of Key Outcomes 



| Outcome 


Definition 


1 1 


Attendance rate 


The total number of days a student was marked present during the school year divided by the 
total number of days the student was listed as enrolled. These data were available consistently 
only for students who attended school for at least one day in the final marking period of the 
year. Thus, the analysis did not include students who dropped out or left the district prior to that 
point. 


1 1 


Chronic 

absenteeism 


Indicates that a student had an attendance rate of 80 percent or lower for the year. 


1 1 


Regular 

attendance 


Indicates that a student had an attendance rate of 90 percent or higher for the year. 


1 1 


Average NCE 


The average Normal Curve Equivalent (NCE) score for students taking the test in a given 
subject area. The normalized test score, which ranges from 1 to 99 with a mean of 50, allows 
for comparison across tests and subjects. 


1 1 


At or above 
grade level 


The percentage of students scoring at or above grade level on the test as indicated by scoring at 
or above the 50th percentile. 


1 1 


In the bottom 
quartile 


The percentage of students scoring at or below the 25th percentile on the test. 


1 1 


Promoted to the 
9th grade 


Classification for a student who was designated in the district’s administrative records as an 
eighth-grader in a given school year and was designated as a ninth-grader in the following 
school year. Students who were not in the district’s administrative records in either year were 
not classified. 



To provide a concrete example in support of the descriptions, this appendix refers throughout to 
the State Standards Assessment (SSA) eighth-grade math test score outcome, measured in 
Normal Curve Equivalents (NCEs ). 4 



4 The Normal Curve Equivalent (NCE) is a way of measuring where a student falls along the normal curve. 
The normalized test scores, which range from 1 to 99 with a mean of 50, allow for comparison across tests and 
subjects. Unlike percentile rank scores, the NCE measurement has an equal interval between scores, which 
means that NCE scores can be averaged to allow for comparisons of groups of students or schools. 
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The Interrupted Time Series Approach 

Estimating Deviations from Baseline for Talent Development Schools 

For this evaluation, outcomes for students enrolled in a given school prior to Talent De- 
velopment implementation were compared with outcomes for students enrolled in the same 
school during the years after implementation began. For most measures of student engagement 
and performance, the analysis focuses on the three years prior to implementation and for up to 
five years after implementation. 5 The three years prior to implementation are referred to as the 
baseline period. The year of implementation and each subsequent year are referred to as follow- 
up years. Differences in student outcomes between the baseline and follow-up periods are re- 
ferred to as deviations from the baseline. 

The key feature of the interrupted time series approach is to project what student en- 
gagement and performance would most likely be without Talent Development. This projection 
extends over one or more years after Talent Development began and is based on measures of 
student engagement and performance during a multiyear pre-Talent Development baseline pe- 
riod. For example, to project into the follow-up period a school’s pattern of math achievement, 
the analysis used the average annual math test scores of eighth-grade students over the three 
baseline years. The equation below specifies the simplest form of a regression model that can be 
used to estimate an interrupted time series from a baseline derived from the three-year average 
at a single school. 6 



r, = A + ±D k FY ll + e, 



k = 1 



where: 



Y : 



SSA math test score for student i 



5 For two schools (School A and School B), only two years of baseline data were available for eighth-grade 
SSA test scores. Therefore, the baseline average for these outcomes is based on only two years of pre-Talent 
Development implementation data. Because these schools began implementation in the 1997-1998 school year, 
there are five years of follow-up data available. For the two schools that began implementation in 1998-1999 
(School C and School D), there are lour years of follow-up data available. For the two schools that began im- 
plementation in 1999-2000 (School E and School F), there are three years of follow-up data available. 

6 It is also possible to project a baseline trend derived from a consistent pattern of year-to-year increases or 
decreases in average test scores in the pre-Talent Development period. This was discounted for the current 
analysis because only three years of pre-Talent Development data are available, leaving only minimal confi- 
dence in an estimate of a consistent year-to-year slope in baseline patterns. For both baseline trend and baseline 
average interrupted time series techniques, see Bloom (2003). 
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FY ki = 1 if student i was a member of the cohort for follow-up Year k, 

and 0 otherwise 

e i - a random error term for student i 

A = a constant term equal to the average SSA math test score of 

eighth-grade students during the baseline years 

D k = the deviation in the average SSA math test score 

from the baseline average A in year k of the follow-up period 
(that is, the Year k deviation from the baseline mean) 

This equation pools data from the baseline and follow-up years and estimates the baseline 
mean and the average deviation from this mean for each year of the follow-up period for a single 
school. Figure 1 shows the unadjusted interrupted time series estimates for one Talent Develop- 
ment school in the district, School F. The triangles plot observed means for each baseline year. 
The solid line represents the baseline average, while the dashed line is the projection of this aver- 
age into the first three Talent Development implementation years. School F began Talent Devel- 
opment implementation in the 1999-2000 school year, so that three years of follow-up data are 
available. The circles plot observed means for each follow-up year. The difference between the 
dashed line and each circle represents the deviation from baseline average for each year of imple- 
mentation. (Note that the years identified on the horizontal axis of this exhibit and Figures 2 
through 6 are presented relative to the first year of Talent Development implementation.) 

Controlling for Changes in Student Characteristics 

In some cases, a Talent Development school (or a comparison school) may experience a 
change in the composition of its student population. For example, neighborhoods may undergo 
demographic changes, or geographic boundaries or rules governing school assignment patterns 
may change. More important, Talent Development may cause a change in the student popula- 
tion, for example, by preventing students from dropping out of school or by reducing the num- 
ber of school transfers (which may keep lower-performing students in school longer). In order 
to help account for systematic changes in the characteristics of student cohorts over time, the 
analysis incorporates individual student characteristics into the model. The equation below 
represents the enhanced regression model for a single school: 

r, =A + f^D,FY ll +^C l X JI +e, 

k = 1 7=1 

where the parameters specified above are the same and: 

X /; = a vector of J background characteristics for student i 
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The Talent Development Evaluation 

Analytic Appendix 
Figure 1 

Average Eighth- Grade SSA Math NCE Scores 
in Talent Development School F, 
Three-Year, Unadjusted, Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The analysis sample includes students not designated as ESOL or special education for whom a test score record 
is available or who were enrolled for at least 145 days during a given school year. 



Cj = the difference in the average eighth-grade SSA math test score over 
time associated with a unit change in background characteristics X 

The capacity of the analysis to control for systematic changes in the characteristics of stu- 
dent cohorts is increased if the X covariates and the outcomes are correlated. For example, sup- 
pose that, in one school, Talent Development increases the percentage of eighth-graders who had 
been retained in a prior grade or who entered middle school with very low achievement levels. 
Such a scenario might occur if Talent Development encouraged students not to transfer or pre- 
vented such students from dropping out of school altogether. Because such students are also less 
likely to be regular school attenders and to score well on the SSA tests, it could appear that Talent 
Development is reducing average student achievement and attendance rates if the analysis does 
not account for this change in the composition of the eighth grade. Thus, it is important to identify 
characteristics that are correlated with key outcomes, such as attendance and academic achieve- 
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ment. This can help disentangle Talent Development’s impact on student achievement from ef- 
fects that are caused by changes in the composition of the eighth-grade cohorts. In this case, the 
following covariates were incorporated into the interrupted times series models: 

OVERAGE = whether the student was overage for her or his current grade, 

indicating that the student been retained in a previous grade. 

RACE = dummy variables indicating whether the student was 

black, white, or of another race 

TEST SCORES = separate variables indicating the student’s fourth-grade reading 
comprehension and math test scores (measured in NCEs) 



Figure 2 shows the adjusted interrupted time series estimates for Talent Development 
School F. As in Figure 1, the triangles plot observed means for each baseline year. The solid 
line represents the baseline average, while the dashed line is the projection of this average into 
the first three Talent Development implementation years. The circles plot observed means for 
each follow-up year. The difference between the dashed line and each circle represents the de- 
viation from baseline average for each year of implementation. 

Accounting for Cohort Effects 

In addition to controlling for changes in student characteristics, the analysis also at- 
tempts to account for cohort effects, which are year-to-year variations in the average engage- 
ment and performance of students as a group or an entire cohort. Because cohort effects reflect 
variation that cannot be adequately explained or controlled for by individual random sampling 
error, the analysis must account for this variation when estimating the standard error of the pro- 
jected baseline average as well as the standard error of the deviations from the baseline in sub- 
sequent follow-up years. If cohort effects are ignored, the standard error of the deviations from 
the baseline will be understated, and their statistical significance will be overstated. 

For example, Figure 3 shows that average SSA eighth-grade math test scores — pooled 
across several schools — varied around the baseline average. The year-to-year variation is the 
source of estimated uncertainty or random error associated with future projections from the 
baseline average. Thus, it is also an additional source of random error associated with the devia- 
tions from the baseline. The more tightly the outcome averages cluster around the baseline av- 
erage, the more confidence can be placed in future projections from this average and, thus, the 
more confidence can be placed in the estimates of the deviations from the baseline. 
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Analytic Appendix 
Figure 2 

Average Eighth- Grade SSA Math NCE Scores 
in Talent Development School F, 
Three-Year, Adjusted, Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The analysis sample includes students not designated as ESOL or special education for whom a test score record 
is available or who were enrolled for at least 145 days during a given school year. 



Cohort effects can be accounted for by adding a random error term v t for each cohort to 
the random error term in the equation above. Adding this error component to the interrupted 
time-series model above yields the following equation: 

Y i =A + ZD k FY ki +ZC j X Ji+ v t +e i 

This equation cannot be estimated using ordinary least squares. This error structure 
represents a form of a hierarchical linear model . 7 Therefore, in order to use comparative inter- 
rupted time series techniques to estimate the effect of Talent Development on student per- 
formance, the interrupted time series model is translated into a multilevel system of equations. 



7 See Raudenbush and Bryk, 2001. 
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The Talent Development Evaluation 



Analytic Appendix 
Figure 3 

Average Eighth-Grade SSA Math Test Scores 
in Early-Implementing Talent Development Schools and Their Comparison Schools, 
for Pre-Talent Development Baseline Years 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The analysis sample includes students not designated as ESOL or special education for whom a test score record 
is available or who were enrolled for at least 145 days during a given school year. 

Results were pooled over six Talent Development middle schools and over six clusters of non-Talent Development 
middle schools. 



In particular, the structure of the analysis can be thought of as having three levels: students 
nested within annual cohorts nested within schools. 

The analysis can be executed using hierarchical linear modeling software. In this case, 
the analysis modeled an equivalent composite equation through the use of the Proc Mixed pro- 
cedure in SAS software . 8 This procedure also allows for calculation of impact estimates and 
corresponding standard errors by school cluster, with each cluster consisting of one Talent De- 

Tor a full description of HLM equations for comparative interrupted time series analyses, see Snipes 
(2003). For more information on using Proc Mixed in the SAS program, see Singer (1998). 
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velopment school and its set of non-Talent Development comparison schools. The average of 
these cluster-by-cluster impacts represents an estimate of the net impact of Talent Development 
on student outcomes. 



Comparison-School Analysis 

Identifying Comparison Schools 

The analysis uses comparison schools to assess the extent to which the baseline and fol- 
low-up patterns of student engagement and performance in Talent Development schools differ 
from the patterns of students in similar schools that do not attempt to implement the Talent De- 
velopment model. The comparison was accomplished by matching each Talent Development 
middle school with one or more schools in the district that served students with similar charac- 
teristics and exhibited a similar pattern of student outcomes during the period before Talent De- 
velopment began. In this way, the non-Talent Development comparison schools can provide a 
good indication of the effects on student engagement and performance that may be caused by 
other policies and events that occur in the district over and above those brought about by Talent 
Development. To get as robust an estimate of these potential effects as possible, the analysis 
sought to identify truly comparable non-Talent Development schools and to include as many 
comparable non-Talent Development schools as possible. 

Comparison schools were selected from among the 27 nonselective, comprehensive 
middle schools in the district that were not implementing Talent Development prior to the 1997- 
1998 school year. The criteria for identifying comparison schools are based on average student 
characteristics and student outcomes over the two years before Talent Development was first 
introduced in the district. Specifically, schools were classified by racial/ethnic composition and 
by math and reading test scores of eighth-grade students averaged over the 1995-1996 and 
1996-1997 school years. Matching focused on eighth-grade characteristics for several reasons. 
First, eighth grade marks the culmination of students’ middle school experiences and the start of 
a critical transition period for young people. Eighth-grade students’ engagement and perform- 
ance are critical indicators of their preparedness for the challenges of transitioning successfully 
to high school. Second, the Talent Development Middle School model makes an effort to pro- 
vide added supports and to upgrade curricula and instruction for all middle school grades. The 
impact of Talent Development on the engagement and performance of eighth-grade students 
represents the cumulative effect of the model for middle schools. Finally, the district places 
heavy emphasis on the SSA, which is given in eighth-grade, as an indicator of student and 
school performance. Therefore, it is important to ensure that, prior to implementation, Talent 
Development and non-Talent Development schools are as similar as possible on the achieve- 
ment levels of eighth-grade students on the SSA math and reading tests. 
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The process of identifying schools that were as comparable as possible to the eventual 
Talent Development middle schools occurred in two steps. The first step was designed to ensure a 
high degree of similarity in the racial/ethnic composition of the eighth-graders. Here, the 38 non- 
selective, comprehensive middle schools in the district were stra t ified into four mutually exclusive 
groups based on their racial and ethnic composition. These groups included schools in which: 

• 90 percent or more of the eighth- grade population were black 

• more than half the eighth-grade population was black, but the racial/ethnic 
composition of the remaining eighth-graders was moderately mixed 

• about 60 percent of the eighth-grade population was of other races/ethnicities 
(predominantly Hispanic) 

• a th ir d or more of the eighth-grade population were white 

All the Talent Development middle schools fell into the first three groups. For each 
Talent Development school, potential comparison schools were limited to those that fell into 
the same group. 

The second step in identifying schools that were comparable to the Talent Development 
schools was examining average SSA eighth-grade test scores in math and reading. In order to 
consider both math and reading test scores at once, the matching process was based on the aver- 
age of each student’s math and reading NCE score. Schools were considered comparable if the 
average eighth-grade composite math and reading score fell within .25 standard deviation of the 
average for a given Talent Development school. 9 This process resulted in groups of 1 to 12 non- 
Talent Development comparison schools for each Talent Development school. 10 Some non- 
Talent Development schools serve as comparison schools for more than one cluster. 

The more similar the two groups of schools were prior to the start of Talent Develop- 
ment, the more likely it is that differences that emerged later can be attributed to the implemen- 
tation of Talent Development. Table 4 provides an indication of the extent to which the match- 
ing process resulted in a group of non-Talent Development schools that was comparable to the 
Talent Development middle schools in the study. The table compares the 11 Talent Develop- 
ment schools with their matched sets of non-Talent Development comparison schools, 

9 The standard deviation of average eighth-grade combined math and reading SSA scores was calculated 
for all 38 nonselective high schools in the district over two years prior to Talent Development implementation 
( 1995-1996 and 1996-1997). Over this period, the average combined test score was 29 NCEs, and the standard 
deviation was 15.2 NCEs. Thus, a .25 standard deviation for the eighth-grade math and reading test score was 
equivalent to 3.8 NCEs. 

l0 Non-Talent Development schools may be included as comparisons for more than one Talent Develop- 
ment School. 
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The Talent Development Evaluation 
Analytic Appendix 
Table 4 



Characteristics of Eighth-Grade Students in Talent Development Schools 
and Non-Talent Development Comparison Schools, 

Averaged Over the Pre-Talent Development Baseline Period 



Characteristic 


Talent 

Development 

Schools 


Non-Talent 

Development 

Schools 


Difference 


Race/ethnicity (%) 


Black 


81.5 


81.8 


-0.3 


White 


4.4 


2.2 


2.1 


Hispanic 


11.2 


14.2 


-3.0 


Other 


2.9 


1.7 


1.2 


Overage for grade a (%) 


21.4 


23.8 


-2.4 


SSA test scores 


Average math and reading (NCE) 


27.0 


26.9 


0.1 


Math 


Average NCE 


25.9 


25.7 


0.2 


In the bottom quartile (%) 


77.5 


77.1 


0.4 


At or above grade level (%) 


6.3 


6.0 


0.3 


Reading 


Average NCE 


28.4 


28.5 


-0.1 


In the bottom quartile (%) 


71.1 


70.4 


0.7 


At or above grade level (%) 


9.0 


9.0 


-0.1 


Attendance rate b 


84.3 


84.8 


-0.5 


Attendance rate of 90% or higher 5 (%) 


47.3 


48.6 


-1.3 


Attendance rate of 80% or lower 5 (%) 


28.5 


26.6 


1.9 


Promoted to 9th grade 0 (%) 


97.5 


97.9 


-0.4 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 11 Talent Development middle schools and 18 non-Talent Development middle 
schools. The analysis sample includes students not designated as ESOL or special education for whom a test score record is available 
or who were enrolled for at least 145 days during a given school year. 

Results in the non-Talent Development columns reflect averages across 1 1 clusters, including both early-implementing and later- 
implementing school clusters, of non-Talent Development schools. Each cluster consisted of 1 to 12 non-Talent Development 
schools. Some non-Talent Development schools were counted in more than one cluster. 

Estimates are not regression-adjusted for students' background characteristics or prior achievement. 

Numbers reflect averages over the three-year period prior to the initial implementation of Talent Development for a given cluster. 

a Typically, students who were overage for grade were retained in the current grade or a prior one. “Overage for grade” means a 
student turned 12 before the start of the 6th grade, 13 before the start of the 7th grade, or 14 before the start of the 8th grade. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of 
days the student was enrolled in a given school year. 

c For the purposes of this analysis, 8th-grade students were considered to have been promoted to the 9th grade if they were listed as 
9th-graders in the district’s administrative data file one year after the current year. Students whose records were not included in the 
data file one year after the current year, for whatever reason, were not in the analysis sample for this outcome. 
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reflecting average characteristics of eighth-graders in each group over the three years prior to 
the implementation of Talent Development in each school cluster. The table indicates that there 
were only modest differences between the Talent Development and non-Talent Development 
comparison schools over the years leading up to Talent Development implementation. The ana- 
lytic strategy described above controls for these initial differences by framing the impacts of 
Talent Development in terms of differences between Talent Development and non-Talent De- 
velopment schools in their deviations from the baseline averages. 

The information presented in Table 4 may mask some year-to-year differences between 
the groups of schools or a trend, upward or downward, that may occur for the non-Talent De- 
velopment schools or for the Talent Development schools. The more that the baseline averages 
remain stable and similar from year to year, the more likely it is that changes in these averages 
are truly caused by Talent Development rather than indicating random spikes or troughs. Fur- 
thermore, the less the variation in baseline trends from year to year, the more likely that these 
trends would continue into the future if Talent Development were not implemented. 

For a specific example of how the variation plays out in terms of outcomes, it is useful 
to refer to Figure 3. The figure shows the year-to-year variation in the SSA math achievement 
outcome for Talent Development schools and their non-Talent Development comparison 
schools in the baseline period. It indicates that the Talent Development schools exhibited simi- 
lar average math achievement as their non-Talent Development counterparts in the baseline pe- 
riod. The figure also indicates that there was some year-to-year variation in the outcome in the 
Talent Development schools and in the comparison schools but no clear slope. 1 1 

Estimating Deviations from Baseline for Comparison Schools 

To compare deviations from baseline for Talent Development and non-Talent Devel- 
opment schools, it was necessary to designate a baseline and follow-up period for each group of 
non-Talent Development comparison schools based on the start of Talent Development imple- 
mentation in their matched Talent Development school. 

As with the Talent Development schools, the interrupted time series approach was ap- 
plied to non-Talent Development schools, cluster by cluster. Figure 4 illustrates the adjusted 
interrupted time series estimates for one group of non-Talent Development comparison schools, 
those matched with Talent Development School F. As in Figures 1 and 2, the triangles plot ob- 



1 1 As noted earlier, it is possible to project a baseline trend derived from a consistent pattern of year-to-year 
increases or decreases in the outcome in the baseline period. This was discounted for the current analysis be- 
cause only three years of pre-Talent Development data are available, which gives minimal confidence in slope 
estimates. 
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served means for each baseline year. In this case, these are the baseline means across the com- 
parison schools in the School F cluster. The solid line represents the baseline average, while the 

The Talent Development Evaluation 

Analytic Appendix 
Figure 4 

Average Eighth- Grade SSA Math NCE Scores 
in Non-Talent Development Comparison Schools in Cluster F, 

Three-Year, Adjusted, Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The analysis sample includes students not designated as ESOL or special education for whom a test score record 
is available or who were enrolled for at least 145 days during a given school year. 



dashed line is the projection of this average into the first three Talent Development implementa- 
tion years. The circles plot observed means for each follow-up year across the comparison 
schools in the School F cluster. The difference between the dashed line and each circle repre- 
sents the deviation from baseline average for each follow-up year. As with Talent Development 
School F, there is improvement in math achievement in the follow-up period as compared with 
the baseline period for this group of comparison schools. Therefore, in this example, impacts 
are driven by the difference in the magnitude of improvement between the Talent Development 
school and the average improvement in its group of comparison schools. 
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Estimating Impacts 

The equations described above were used to generate estimated deviations from the 
baseline average for each Talent Development school and for each Talent Development 
school’s matched group of non-Talent Development schools. 12 In this report, the analysis fo- 
cuses on six middle schools in the district that began working with Talent Development in the 
1997-1998, 1998-1999, and 1999-2000 school years; these are referred to throughout the report 
as “early-implementing schools.” 13 Table 5 presents the results from an analysis of eighth-grade 
SSA math achievement in these six schools and the comparison schools in their school cluster. 
These results are presented cluster by cluster in order to illustrate the variability of results from 
year to year and school to school, as well as to help illustrate how impacts are pooled across 
schools. Impacts for individual schools may not be reliable. 

The pooled estimates maximize the reliability of the impact estimates, because esti- 
mates for any one school or cluster may be anomalous. In this way, the analysis can assess the 
likelihood that a nonzero impact was due to chance. In general, the larger the number of schools 
that exhibit a nonzero impact, the higher the likelihood that the analysis can detect real changes 
in student engagement and performance that were produced by Talent Development. 

On the first page of Table 5, the five columns of numbers at the left show average math 
achievement (in NCEs) for the Talent Development schools and the deviation from baseline 
that each average represents for each year of implementation. The five columns of numbers at 
the right show these results for the non-Talent Development comparison schools in each school 
cluster. Under the School F heading, for example, the three columns at the left of the first row 
show average math achievement in each of three implementation years for Talent Development 
School F; the three columns at the right show the same information averaged across the non- 
Talent Development schools in this cluster. (This information for School F and its cluster of 
comparison schools is also illustrated in Figures 2 and 4, respectively.) 



12 Note that, because some non-Talent Development schools served as comparison schools for more than 
one Talent Development School, multiple estimates were obtained for these schools. Furthermore, different 
baseline averages were estimated for some non-Talent Development schools that served as comparison schools 
for Talent Development schools that began implementation in different years. 

13 The report also includes limited analysis for the first year of implementation in another five middle 
schools that began implementing the model in the district in the 2001-2002 school year (referred to as “later- 
implementing schools”). 
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The Talent Development Evaluation 
Analytic Appendix 
Table 5 

Year-by-Year Levels and Impacts for SSA Math NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 



School Cluster 




Talent Development Schools 






Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















8 th-grade average NCE 


24.8 


28.1 


30.7 


34.8 


35.0 


27.2 


28.4 


29.0 


28.8 


29.9 


Deviation from baseline 


0.8 


4.1 


6.6 ** 


10.8 *** 


] ] Q *** 


1.6 


29 ** 


3 4 *** 


3 2 *** 


4 4 *** 


School B 






















8 th-grade average NCE 


29.0 


30.6 


34.5 


34.0 


33.1 


27.6 


31.6 


31.5 


30.9 


30.8 


Deviation from baseline 


0.3 


2.0 


5.9 * 


5.3 


4.5 


- 0.2 


3.7 * 


3.6 * 


3.0 


2.9 


School C 






















8 th-grade average NCE 


23.7 


23.8 


25.7 


28.0 




22.8 


26.0 


25.1 


27.7 




Deviation from baseline 


4.2 * 


4.3 * 


6.2 ** 


8.6 *** 




2.1 ** 


5 3 *** 


4 4 *** 


7 q *** 




School D 






















8 th-grade average NCE 


25.8 


25.9 


26.8 


30.8 




22.5 


26.5 


25.4 


26.8 




Deviation from baseline 


3.2 * 


3.3 * 


4.2 ** 


g 9 *** 




- 0.2 


3.7 ** 


2.7 ** 


4 | *** 




School E 






















8 th-grade average NCE 


26.1 


26.3 


30.3 






27.2 


27.1 


28.7 






Deviation from baseline 


0.7 


0.9 


4.9 * 






1.4 


1.3 


3 Q *** 






School F 






















8 th-grade average NCE 


27.2 


29.1 


27.4 






25.6 


25.1 


26.9 






Deviation from baseline 


6.4 ** 


g 3 *** 


6.6 ** 






3 1 *** 


2.6 *** 


4 4 *** 









All early-implementing schools 
8 th-grade average NCE 
Deviation from baseline 


26.1 
2.6 ** 


27.3 

3 g *** 


29.2 
5.7 *** 


31.9 

g 2 *** 


34.1 

7 7 *** 


25.5 
1.3 *** 


27.4 
3 2 *** 


27.8 
3.6 *** 


28.6 

4 3 *** 


30.4 
3.6 *** 



(continued) 
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The Talent Development Evaluation 
Analytic Appendix 
Table 5 (continued) 



Year-by-Year Levels and Impacts for SSA Math NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















8 th-grade average NCE 
Deviation from baseline 


-0.9 


1.2 


3.2 


7.6 ** 


6.6 ** 


-0.06 


0.09 


0.22 


0.53 ** 


0.46 ** 


School B 






















8 th-grade average NCE 
Deviation from baseline 


0.6 


-1.7 


2.3 


2.3 


1.6 


0.04 


- 0.12 


0.16 


0.16 


0.11 


School C 






















8 th-grade average NCE 
Deviation from baseline 


2.1 


-0.9 


1.8 


1.6 




0.15 


-0.07 


0.13 


0.11 




School D 






















8 th-grade average NCE 
Deviation from baseline 


3.4 


-0.4 


1.5 


4.1 * 




0.24 


-0.03 


0.11 


0.29 * 




School E 






















8 th-grade average NCE 
Deviation from baseline 


-0.7 


-0.4 


1.9 






-0.05 


- 0.02 


0.13 






School F 






















8 th-grade average NCE 
Deviation from baseline 


3.3 


5.7 ** 


2.2 






0.23 


0.40 


** 0.16 








All early-implementing schools 
8 th-grade average NCE 






















Deviation from baseline 


1.3 


0.6 


2.2 


3 9 *** 


4.1 * 


0.09 


0.04 


0.15 


q 27 *** 


0.28 * 



(continued) 
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Analytic Appendix Table 5 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent 
Development schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, 
Year 3, Year 4. and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools 
and non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or 
data were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * 
= 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the 
bottom of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 




The second row under the School F heading shows the deviation from the baseline av- 
erage represented by the difference in average math achievement in each follow-up year and 
average math achievement over the three years prior to Talent Development (the baseline aver- 
age). For example, for Talent Development School F, the average deviation from baseline in 
Year 1 is 6.4 NCEs. This indicates that eighth-grade SSA math scores in School F increased by 
an average of 6.4 NCEs during the first year of Talent Development implementation. In the 
same row, at the right, the numbers show that math scores increased by an average of 3.1 NCEs 
in the non-Talent Development schools in the same cluster over the same time period. There- 
fore, the estimated impact of Talent Development at School F is 3.3 NCEs, or the difference 
between the deviation from baseline for the Talent Development school and the average devia- 
tion from baseline for its comparison schools. This first-year impact is shown on the second 
page of Table 5, also under the School F heading. The impact corresponds to an effect size of 
0.23. 14 Again, estimated impacts for individual schools may not be reliable and are given here 
only to illustrate how estimates were pooled to arrive at the impact across all schools. 

The bottom two rows of numbers on both pages of Table 5 show results that have been 
pooled across all six school clusters included in the analysis. The cross-cluster average math 
achievement and deviation from baseline for each follow-up year were obtained by computing a 
simple mean across the six early-implementing Talent Development schools and the six non- 
Talent Development comparison group averages. Combined standard errors were computed for 
each of these means accordingly. 15 For example, on average, the deviation from baseline in the 
follow-up period for Talent Development schools was 2.6 NCEs. The average deviation from 
baseline for non-Talent Development schools was 1.3 NCEs. Therefore, the impact of Talent 
Development on the average eighth-grade SSA math achievement is an increase of 1.3 NCEs. 
This impact is not statistically significant and corresponds to an effect size of 0.09. 

Pooled estimates maximize the reliability of the impact estimates. However, it should 
be noted that follow-up data are available for all six early-implementing school clusters only for 
the first three years of implementation. Findings for these first three years have the greatest sta- 
tistical power and show the most robust indication of Talent Development’s preliminary impact 
on student performance and attendance. Also, indications of statistical significance, 16 which de- 



l4 Effect sizes show each impact as an proportion of the comparison-group student-level standard deviation 
for each outcome. In this report, the standard deviations used to calculate effect sizes are based on two years of 
pre-Talent Development implementation data from 11 Talent Development schools and 18 non-Talent Devel- 
opment comparison schools. 

15 The formula for standard errors for an average of adjusted means was used. The analysis is not able to 
account for the fact that some comparison schools were used in more than one cluster. 

l6 Statistical significance is a measure of the degree of certainty one may have that some nonzero deviation 
from the baseline average actually occurred. For example, if an impact estimate is statistically significant, then 
one may conclude with some confidence that the program really had an effect. If an impact estimate is not sta- 

(continued) 



- 25 - 




pend in part on sample size, may be achieved with impacts of smaller magnitude in the first 
three years as compared with impacts in Years 4 and 5, which include fewer schools. Similarly, 
average deviations from baseline for Talent Development schools include at most 6 schools, 
while average deviations from baseline for non-Talent Development schools include up to 18 
schools. Again, in this instance, statistical significance may be achieved with smaller deviations 
from baseline for non-Talent Development comparison schools as compared with deviations 
from baseline for Talent Development schools. 

Figure 5 provides a graphic representation of the findings presented at the bottom of 
Table 5. The top panel of the figure shows the baseline average and the deviation from the pro- 
jected baseline average for the Talent Development schools. The bottom panel presents this in- 
formation for the non-Talent Development schools. The solid line in each panel represents the 
baseline average eighth-grade SSA math test score, and the dashed line represents the projection 
of that average into the postimplementation follow-up period. The triangles show the average 
SSA math test score in each year prior to the start of Talent Development implementation aver- 
aged across all six school clusters. 

The circles in each part of the figure represent the average SSA math test score in the 
first three years of Talent Development implementation across six school clusters. The differ- 
ences between the dashed lines and the circles represent deviations from the baseline for Talent 
Development (in the upper panel) and non-Talent Development schools (in the lower panel). 
Again, it is the difference in these two deviations that represents the impact of Talent Develop- 
ment. Figure 6 shows the difference in deviations more clearly. For each follow-up year, the 
black bar represents the deviation from baseline averaged across six Talent Development 
schools, and the white bar represents the deviation from baseline average across six clusters of 
non-Talent development comparison schools. 

For this study, this process of estimating and pooling impacts across school clusters 
was repeated for several student outcomes, including reading achievement, attendance rates, 
and grade-level promotion. For some outcomes and some follow-up years, data for all six 
school clusters are not available. In most cases, data for four school clusters are available to es- 
timate the impact of Talent Development in the fourth year of implementation, and data from 
two school clusters are available to estimate the impact in the fifth year of implementation. The 
number of school clusters included in pooled impact estimates for each year is noted in the text, 
tables, and footnotes of the main report. 

tistically significant, then the nonzero estimate is more likely to be the product of chance or random variation in 
the averages that were calculated across the schools and years under study. Unless otherwise noted, the devia- 
tions from baseline averages and Talent Development impacts discussed in this report are statistically signifi- 
cant at the 10 percent level or lower. This means that there is no more than a 10 percent probability that the 
difference resulted only from chance or random variation 
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The Talent Development Evaluation 

Analytic Appendix 
Figure 5 



Average Eighth-Grade SSA Math NCE Scores 
in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Three-Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: The analysis sample includes students not designated as ESOL or special education for whom a test score record is 
available or who were enrolled for at least 145 days during a given school year. 

Results were pooled over six Talent Development middle schools and over six groups of non-Talent Development middle 
schools. 
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The Talent Development Evaluation 
Analytic Appendix 
Figure 6 

Impacts on SSA Math NCE Scores 

for Eighth-Grade Students in Early-Implementing Talent Development Schools, 
Three- Year Follow-Up Results 
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SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development 
middle schools. The analysis sample includes students not designated as ESOL or special education for whom a test score 
record is available or who were enrolled for at least 145 days during a given school year. 

Results are pooled over six Talent Development Schools and six clusters of non-Talent Development comparison 
schools. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in math NCE points from the three-year pre-implementation baseline average to each follow-up year. 

The impact was calculated as the difference in deviations from the baseline average between Talent Development 
schools and non-Talent Development comparison schools. 

A two-tailed t-test was applied to the impacts. Standard errors were adjusted to account for cohort effects. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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The Talent Development Evaluation 
TR Table A.l 



Year-by-Year Levels and Impacts for SSA Math NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 











Outcome Levels Compared with Baseline Average 












Talent Development Schools 






Non-Talent Development Schools 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















8 th-grade average NCE 


24.8 


28.1 


30.7 


34.8 


35.0 


27.2 


28.4 


29.0 


28.8 


29.9 


Deviation from baseline 


0.8 


4.1 


6.6 ** 


10.8 *** 


H Q *** 


1.6 


2.9 ** 


3 4 *** 


3 2 *** 


4 4 *** 


School B 






















8 th-grade average NCE 


29.0 


30.6 


34.5 


34.0 


33.1 


27.6 


31.6 


31.5 


30.9 


30.8 


Deviation from baseline 


0.3 


2.0 


5.9 * 


5.3 


4.5 


- 0.2 


3.7 * 


3.6 * 


3.0 


2.9 


School C 






















8 th-grade average NCE 


23.7 


23.8 


25.7 


28.0 




22.8 


26.0 


25.1 


27.7 




Deviation from baseline 


4.2 * 


4.3 * 


6.2 ** 


8.6 *** 




2.1 ** 


5.3 *** 


4 4 *** 


7 Q *** 




School D 






















8 th-grade average NCE 


25.8 


25.9 


26.8 


30.8 




22.5 


26.5 


25.4 


26.8 




Deviation from baseline 


3.2 * 


3.3 * 


4.2 ** 


8,2 *** 




- 0.2 


3.7 ** 


2.7 ** 


4 1 *** 




School E 






















8 th-grade average NCE 


26.1 


26.3 


30.3 






27.2 


27.1 


28.7 






Deviation from baseline 


0.7 


0.9 


4.9 * 






1.4 


1.3 


3 0 *** 






School F 






















8 th-grade average NCE 


27.2 


29.1 


27.4 






25.6 


25.1 


26.9 






Deviation from baseline 


6.4 ** 


8 3 *** 


6.6 ** 






3 1 *** 


2.6 *** 


4 4 *** 








All early-implementing schools 






















8 th-grade average NCE 


26.1 


27.3 


29.2 


31.9 


34.1 


25.5 


27.4 


27.8 


28.6 


30.4 


Deviation from baseline 


2.6 ** 


3 g *** 


57 *** 


8,2 *** 


y 7 *** 


1.3 *** 


3 2 *** 


3.6 *** 


4 3 *** 


3.6 *** 



(continued) 
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The Talent Development Evaluation 
TR Table A.l (continued) 



Year-by-Year Levels and Impacts for SSA Math NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















8 th-grade average NCE 
Deviation from baseline 


-0.9 


1.2 


3.2 


7.6 ** 


6.6 ** 


-0.06 


0.09 


0.22 


0.53 ** 


0.46 ** 


School B 






















8 th-grade average NCE 
Deviation from baseline 


0.6 


-1.7 


2.3 


2.3 


1.6 


0.04 


- 0.12 


0.16 


0.16 


0.11 


School C 






















8 th-grade average NCE 
Deviation from baseline 


2.1 


-0.9 


1.8 


1.6 




0.15 


-0.07 


0.13 


0.11 




School D 






















8 th-grade average NCE 
Deviation from baseline 


3.4 


-0.4 


1.5 


4.1 * 




0.24 


-0.03 


0.11 


0.29 * 




School E 






















8 th-grade average NCE 
Deviation from baseline 


-0.7 


-0.4 


1.9 






-0.05 


- 0.02 


0.13 






School F 






















8 th-grade average NCE 
Deviation from baseline 


3.3 


5.7 ** 


2.2 






0.23 


0.40 


** 0.16 








All early-implementing schools 
8 th-grade average NCE 






















Deviation from baseline 


1.3 


0.6 


2.2 


3 9 *** 


4.1 * 


0.09 


0.04 


0.15 


0.27 *** 


0.28 * 



(continued) 
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TR Table A.l (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 1 45 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table A.2 

Year-by-Year Levels and Impacts for SSA Math Scores At or Above Grade Level 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 







Talent Development Schools 






Non-Talent Development Schools 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















At or above grade level (%) 


3.5 


7.2 


7.4 


17.3 


14.0 


7.0 


10.1 


9.3 


6.9 


8.0 


Deviation from baseline 


- 0.6 


3.1 


3.3 


13.2 *** 


9 9 ** 


1.7 


4 § *** 


4 o *** 


1.5 


2.7 * 


School B 






















At or above grade level (%) 


10.1 


12.4 


17.8 


13.1 


13.4 


6.9 


12.1 


10.7 


8.9 


8.0 


Deviation from baseline 


- 0.8 


1.5 


6.9 * 


2.2 


2.5 


-0.4 


4.8 * 


3.4 


1.6 


0.7 


School C 






















At or above grade level (%) 


4.7 


1.8 


4.0 


3.4 




2.2 


6.2 


4.4 


3.9 




Deviation from baseline 


1.7 


- 1.2 


1.0 


0.4 




0.2 


4 2 *** 


2.5 ** 


2.0 * 




School D 






















At or above grade level (%) 


3.3 


4.7 


5.3 


11.7 




5.4 


4.0 


2.7 


5.0 




Deviation from baseline 


0.8 


2.2 


2.8 


9 \ *** 




2.2 


0.8 


-0.5 


1.8 




School E 






















At or above grade level (%) 


6.9 


3.8 


3.4 






7.8 


5.4 


7.4 






Deviation from baseline 


2.2 


-0.9 


-1.4 






1.4 


- 1.0 


1.0 






School F 






















At or above grade level (%) 


4.0 


5.0 


5.8 






6.1 


3.9 


3.4 






Deviation from baseline 


1.5 


2.5 


3.3 






3 0 *** 


0.8 


0.3 








All early-implementing schools 






















At or above grade level (%) 


5.4 


5.8 


7.3 


11.4 


13.7 


5.9 


7.0 


6.3 


6.2 


8.0 


Deviation from baseline 


0.8 


1.2 


2.7 ** 


6.2 *** 


6.2 ** 


1.4 ** 


2.4 *** 


1.8 *** 


1.7 * 


1.7 



(continued) 
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The Talent Development Evaluation 
TR Table A.2 (continued) 

Year-by-Year Levels and Impacts for SSA Math Scores At or Above Grade Level 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

At or above grade level (%) 
Deviation from baseline 


-2.3 


-1.7 


-0.7 


11.6 ** 


7.2 


-0.10 


-0.08 


-0.03 


0.53 ** 


0.33 


School B 

At or above grade level (%) 
Deviation from baseline 


-0.4 


-3.3 


3.5 


0.6 


1.8 


-0.02 


-0.15 


0.16 


0.03 


0.08 


School C 

At or above grade level (%) 
Deviation from baseline 


1.5 


-5.5 * 


-1.5 


-1.6 




0.07 


-0.25 * 


-0.07 


-0.07 




School D 

At or above grade level (%) 
Deviation from baseline 


-1.5 


1.4 


3.3 


7.3 * 




-0.07 


0.06 


0.15 


0.33 




School E 

At or above grade level (%) 
Deviation from baseline 


0.8 


0.1 


-2.3 






0.04 


0.00 


-0.11 






School F 

At or above grade level (%) 
Deviation from baseline 


-1.5 


1.6 


3.0 






-0.07 


0.07 


0.14 








All early-implementing schools 
At or above grade level (%) 
Deviation from baseline 


-0.6 


-1.2 


0.9 


4.5 ** 


4.5 


-0.03 


-0.06 


0.04 


0.20 ** 


0.20 



(continued) 
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TR Table A.2 (continued) 



SOURCE: MDRC calculations from individual students' school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table A.3 

Year-by-Year Levels and Impacts for SSA Math Scores in the Bottom Quartile 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















In the bottom quartile (%) 


80.4 


67.3 


68.9 


53.5 


58.0 


74.1 


68.1 


69.7 


73.1 


68.5 


Deviation from baseline 


-3.0 


-16.1 * 


-14.4 * 


-29.9 *** 


-25.3 *** 


-4.6 


-10.5 *** 


-9.0 *** 


-5.6 * 


-10.2 *** 


School B 






















In the bottom quartile (%) 


68.2 


61.4 


57.9 


56.3 


57.0 


72.8 


60.9 


62.7 


66.2 


65.2 


Deviation from baseline 


-4.0 


-10.7 


-14.2 


-15.8 * 


-15.1 * 


-0.8 


-12.7 ** 


-10.9 * 


-7.4 


-8.4 


School C 






















In the bottom quartile (%) 


86.5 


79.3 


81.8 


78.7 




84.3 


78.0 


79.8 


75.7 




Deviation from baseline 


-6.2 


-13.4 ** 


-11.0 * 


-14.0 ** 




-3.2 


_9 4 *** 


-7.6 *** 


-11.8 *** 




School D 






















In the bottom quartile (%) 


75.0 


77.0 


76.8 


69.0 




82.6 


78.4 


81.6 


78.5 




Deviation from baseline 


-12.1 * 


-10.1 * 


-10.3 * 


-18.1 *** 




-0.7 


-4.9 


-1.7 


-4.7 




School E 






















In the bottom quartile (%) 


78.1 


77.0 


69.7 






74.3 


77.4 


71.2 






Deviation from baseline 


0.1 


-0.9 


-8.2 






-2.0 


1.2 


-5.0 ** 






School F 






















In the bottom quartile (%) 


79.2 


70.8 


78.5 






78.8 


81.1 


77.9 






Deviation from baseline 


-7.3 


-15.7 ** 


-8.0 






_4 g *** 


-2.4 


-5.6 *** 








All early-implementing schools 






















In the bottom quartile (%) 


77.9 


72.1 


72.3 


64.4 


57.5 


77.8 


74.0 


73.8 


73.4 


66.8 


Deviation from baseline 


-5.4 * 


-1 1.2 *** 


-1 1.0 *** 


-19.5 *** 


-20.2 *** 


_2.7 ** 


-6.5 *** 


-6.7 *** 


-7.4 *** 


-9.3 *** 



(continued) 
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The Talent Development Evaluation 
TR Table A.3 (continued) 

Year-by-Year Levels and Impacts for SSA Math Scores in the Bottom Quartile 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 Year 5 


Year 1 


Year 2 


Year 3 


Year 4 Year 5 


School A 


















In the bottom quartile (%) 
Deviation from baseline 


1.5 


-5.5 


-5.4 


-24.3 *** -15.1 * 


0.04 


-0.14 


-0.14 


-0.61 *** -0.38 * 


School B 


















In the bottom quartile (%) 
De viation from baseline 


-3.2 


2.0 


-3.3 


oo 

4^ 

p's 

bo 


-0.08 


0.05 


-0.08 


-0.21 -0.17 


School C 


















In the bottom quartile (%) 
Deviation from baseline 


-3.0 


-4.0 


-3.3 


-2.2 


-0.08 


-0.10 


-0.08 


-0.06 


School D 


















In the bottom quartile (%) 
Deviation from baseline 


-11.4 


-5.2 


-8.7 


-13.4 * 


-0.29 


-0.13 


-0.22 


-0.34 * 


School E 


















In the bottom quartile (%) 
Deviation from baseline 


2.1 


-2.1 


-3.2 




0.05 


-0.05 


-0.08 




School F 


















In the bottom quartile (%) 
Deviation from baseline 


-2.6 


-13.3 ** 


-2.4 




-0.06 


-0.34 ** 


-0.06 



























All early-implementing schools 
In the bottom quartile (%) 
Deviation from baseline 


-2.8 


-4.7 


-4.4 


-12.1 *** 


-10.9 


-0.07 


-0.12 


-0.11 


-0.30 *** 


-0.28 



(continued) 
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TR Table A.3 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table A.4 

Year-by-Year Levels and Impacts for SSA Reading NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















8th-grade average NCE 


32.4 


32.7 


31.6 


33.1 


34.4 


30.5 


29.0 


30.7 


30.8 


32.0 


Deviation from baseline 


1.1 


1.3 


0.2 


1.7 


3.0 


0.6 


-0.9 


0.8 


0.8 


2.1 ** 


School B 






















8th-grade average NCE 


33.8 


35.8 


35.9 


35.3 


38.0 


32.3 


31.1 


32.9 


33.2 


33.3 


Deviation from baseline 


0.1 


2.1 


2.2 


1.6 


4.3 


-1.3 


-2.5 


-0.7 


-0.4 


-0.3 


School C 






















8th-grade average NCE 


23.7 


26.7 


24.0 


25.7 




26.0 


27.4 


24.6 


30.5 




Deviation from baseline 


0.5 


3.5 


0.8 


2.5 




0.9 


23** 


-0.5 


5.5 *** 




School D 






















8th-grade average NCE 


23.1 


28.5 


27.7 


29.2 




27.7 


24.0 


25.0 


25.9 




Deviation from baseline 


-3.4 


2.1 


1.3 


2.7 




1.5 


-2.2 


-1.2 


-0.3 




School E 






















8th-grade average NCE 


27.8 


29.6 


30.1 






29.0 


29.1 


30.9 






Deviation from baseline 


-3.4 


-1.5 


-1.0 






0.4 


0.5 


2 3 *** 






School F 






















8th-grade average NCE 


26.2 


30.2 


26.3 






27.5 


26.1 


29.9 






Deviation from baseline 


1.3 


5.4 * 


1.4 






1.0 


-0.4 


3 4 *** 







All early-implementing schools 

8th-grade average NCE 27.8 30.6 29.3 30.8 36.2 28.8 27.8 29.0 30.1 32.7 

Deviation from baseline -0.6 2.1 ** 0.8 2.1 3.7 ** 0.5 -0.5 0.7 1.4 ** 0.9 



(continued) 
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The Talent Development Evaluation 
TR Table A.4 (continued) 

Year-by-Year Levels and Impacts for SSA Reading NCE Scores 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 



Impact 



Impact Effect Size 



Year 1 Year 2 Year 3 Year 4 Year 5 Year 1 Year 2 Year 3 Year 4 Year 5 



School A 

8th-grade average NCE 

Deviation from baseline 0.5 

School B 

8th-grade average NCE 

Deviation from baseline 1 .4 

School C 

8th-grade average NCE 

Deviation from baseline -0.4 

School D 

8th-grade average NCE 

Deviation from baseline -4.9 

School E 

8th-grade average NCE 
Deviation from baseline -3.8 

School F 

8th-grade average NCE 
Deviation from baseline 0.3 



2.2 -0.6 0.8 0.9 

4.7 3.0 2.0 4.6 

1.2 1.3 -2.9 

4.3 2.5 3.0 

-2.0 -3.4 

5.8 * -2.0 



0.03 



0.09 



-0.03 



-0.30 



0.14 



0.29 



0.07 



0.27 



-0.04 0.05 0.06 



0.19 0.12 0.29 

0.08 -0.19 



0.16 0.19 



-0.24 -0.13 -0.21 

0.02 0.36 * -0.12 

























All early-implementing schools 
8th-grade average NCE 
Deviation from baseline 


-1.1 


2.7 ** 


0.1 


0.7 


2.8 


-0.07 


0.17 ** 


0.01 


0.05 


0.17 



(continued) 
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TR Table A.4 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table A.5 



Year-by-Year Levels and Impacts for SSA Reading Scores At or Above Grade Level 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 







Talent Development Schools 






Non-Talent Development Schools 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















At or above grade level (%) 


10.8 


13.9 


10.6 


13.2 


10.9 


9.4 


8.7 


12.1 


10.1 


9.9 


Deviation from baseline 


0.2 


3.4 


0.1 


2.7 


0.3 


-0.6 


-1.3 


2.0 


0.0 


-0.2 


School B 






















At or above grade level (%) 


16.0 


21.2 


22.5 


15.4 


18.6 


10.3 


10.8 


14.7 


14.0 


10.6 


Deviation from baseline 


0.4 


5.6 


6.9 


-0.2 


3.0 


-4.9 


-4.4 


-0.5 


-1.2 


-4.6 


School C 






















At or above grade level (%) 


5.2 


9.5 


3.4 


1.6 




6.5 


8.8 


3.7 


6.5 




Deviation from baseline 


2.6 


7.0 ** 


0.8 


-1.0 




0.6 


3.0 ** 


-2.2 * 


0.6 




School D 






















At or above grade level (%) 


2.4 


00 

bo 


5.6 


9.5 




7.2 


3.8 


4.0 


4.0 




Deviation from baseline 


-5.6 ** 


0.7 


-2.5 


1.4 




-0.6 


-4.1 ** 


* 

* 

OO 

cn 


-3.8 ** 




School E 






















At or above grade level (%) 


11.7 


8.0 


6.2 






10.3 


7.8 


8.5 






Deviation from baseline 


0.0 


-3.7 


-5.5 * 






2.1 * 


-0.3 


0.3 






School F 






















At or above grade level (%) 


5.9 


6.8 


3.9 






8.4 


4.6 


6.3 






Deviation from baseline 


1.0 


1.9 


-1.0 






1.8 * 


-2.1 ** 


-0.3 







All early-implementing schools 

At or above grade level (%) 8.7 11.4 8.7 9.9 14.7 8.7 7.4 8.2 8.6 10.2 

Deviation from baseline -0.2 2.5 -0.2 0.7 1.7 -0.3 -1.5 * -0.7 -1.1 -2.4 



(continued) 
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The Talent Development Evaluation 
TR Table A.5 (continued) 



Year-by-Year Levels and Impacts for SSA Reading Scores At or Above Grade Level 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

At or above grade level (%) 
Deviation from baseline 


0.9 


4.7 


-1.9 


2.7 


0.5 


0.03 


0.16 


-0.07 


0.09 


0.02 


School B 

At or above grade level (%) 
Deviation from baseline 


5.3 


10.0 


7.4 


1.0 


7.6 


0.18 


0.35 


0.26 


0.03 


0.26 


School C 

At or above grade level (%) 
Deviation from baseline 


2.0 


4.0 


3.0 


-1.6 




0.07 


0.14 


0.10 


-0.06 




School D 

At or above grade level (%) 
Deviation from baseline 


-5.0 


4.7 


1.3 


5.2 * 




-0.17 


0.16 


0.05 


0.18 




School E 

At or above grade level (%) 
Deviation from baseline 


-2.1 


-3.4 


-5.8 * 






-0.07 


-0.12 


-0.20 * 






School F 

At or above grade level (%) 
Deviation from baseline 


-0.7 


4.0 


-0.7 






-0.03 


0.14 


-0.02 








All early-implementing schools 
At or above grade level (%) 
Deviation from baseline 


0.0 


4.0 ** 


0.6 


1.8 


4.0 


0.00 


0.14 


** 0.02 


0.06 


0.14 



(continued) 
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TR Table A.5 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 




- 50 - 



The Talent Development Evaluation 
TR Table A.6 



Year-by-Year Levels and Impacts for SSA Reading Scores in the Bottom Quartile 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 



School Cluster 




Talent Development Schools 






Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















In the bottom quartile (%) 


64.9 


65.2 


64.7 


57.3 


59.2 


65.0 


71.5 


67.5 


66.6 


64.8 


De viation from baseline 


0.3 


0.5 


0.0 


-7.4 


-5.5 


-2.2 


4.3 * 


0.3 


-0.6 


-2.4 


School B 






















In the bottom quartile (%) 


57.0 


54.0 


51.6 


51.4 


46.0 


60.3 


68.6 


63.5 


58.2 


59.4 


De viation from baseline 


-2.1 


-5.1 


-7.6 


-7.7 


-13.1 * 


2.2 


10.5 ** 


5.4 


0.1 


1.3 


School C 






















In the bottom quartile (%) 


81.9 


77.9 


89.3 


83.4 




77.3 


75.3 


80.1 


68.7 




De viation from baseline 


0.9 


-3.1 


8.3 


2.4 




0.2 


-1.8 


3.0 


_g 4 *** 


School D 






















In the bottom quartile (%) 


84.7 


70.7 


74.7 


71.7 




72.6 


79.2 


77.6 


81.3 




De viation from baseline 


10.3 


-3.6 


0.3 


-2.7 




-1.8 


4.7 


3.2 


6.9 




School E 






















In the bottom quartile (%) 


75.7 


66.9 


72.1 






71.3 


71.6 


68.7 






Deviation from baseline 


9.9 


1.2 


6.4 






0.9 


1.2 


-1.7 






School F 






















In the bottom quartile (%) 


75.6 


75.0 


78.3 






74.9 


78.0 


70.8 






De viation from baseline 


-1.2 


-1.8 


1.5 






-0.2 


2.9 


-4.3 ** 







All early-implementing schools 

In the bottom quartile (%) 73.3 68.3 71.8 66.0 52.6 70.2 74.0 71.4 68.7 62.1 

Deviation from baseline 3.0 -2.0 1.5 -3.8 -9.3* -0.2 3.6*** 1.0 -0.5 -0.6 



(continued) 
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The Talent Development Evaluation 
TR Table A.6 (continued) 

Year-by-Year Levels and Impacts for SSA Reading Scores in the Bottom Quartile 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















In the bottom quartile (%) 
Deviation front baseline 


2.5 


-3.7 


-0.3 


-6.8 


-3.1 


0.06 


-0.08 


-0.01 


-0.15 


-0.07 


School B 






















In the bottom quartile (%) 
Deviation front baseline 


-4.3 


-15.6 * 


-13.0 


-7.8 


-14.4 * 


-0.09 


-0.34 * 


-0.28 


-0.17 


-0.32 * 


School C 






















In the bottom quartile (%) 
Deviation front baseline 


0.7 


-1.3 


5.3 


10.8 




0.01 


-0.03 


0.12 


0.24 




School D 






















In the bottom quartile (%) 
Deviation front baseline 


12.1 


-8.4 


-2.9 


-9.6 




0.27 


-0.18 


-0.06 


-0.21 




School E 






















In the bottom quartile (%) 
Deviation from baseline 


9.1 


0.0 


8.1 






0.20 


0.00 


0.18 






School F 






















In the bottom quartile (%) 
Deviation from baseline 


-1.0 


-4.7 


5.7 






-0.02 


-0.10 


0.13 








All early-implementing schools 
In the bottom quartile (%) 






















Deviation front baseline 


3.2 


-5.6 * 


0.5 


-3.3 


-8.8 


0.07 


-0.12 * 


0.01 


-0.07 


-0.19 



(continued) 
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TR Table A.6 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4. and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table A.7 

Year-by-Year Levels and Impacts for Attendance Rate 

for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 

School Cluster Year 1 Year 2 Year 3 Year 4 Year 5 Year 1 Year 2 Year 3 Year 4 Year 5 

School A 



8 th grade attendance rate (%) 


84.7 


87.0 


86.9 


87.7 


87.0 


86.5 


87.2 


86.1 


86.3 


86.7 


Deviation from baseline 


0.4 


2.7 


2.6 


3.4 


2.6 


2.0 ** 


2 7 *** 


j 7 ** 


* 

* 

oo 


2 2 *** 


School B 






















8 th grade attendance rate (%) 


88.7 


90.6 


89.6 


88.8 


88.4 


87.8 


87.3 


87.0 


87.4 


89.4 


Deviation from baseline 


1.5 


3.5 


2.5 


1.6 


1.3 


2.1 


1.6 


1.3 


1.7 


3.7 ** 


School C 






















8 th grade attendance rate (%) 


83.5 


83.6 


82.9 


81.6 




85.4 


85.9 


84.8 


85.0 




Deviation from baseline 


4.6 * 


4.8 * 


4.1 


2.8 




2.0 * 


2.5 ** 


1.4 


1.7 


*** 


School D 






















8 th grade attendance rate (%) 


90.4 


86.8 


85.6 


84.0 




82.3 


83.0 


83.3 


84.0 




Deviation from baseline 


6.4 ** 


2.8 


1.6 


0.0 




1.1 


1.9 


2.1 


2.9 




School E 






















8 th grade attendance rate (%) 


87.0 


90.1 


91.9 






85.5 


85.5 


85.5 






Deviation from baseline 


2.0 


5.1 ** 


6.9 *** 






-0.4 


-0.4 ** 


_Q 4 *** 







School F 

8 th grade attendance rate (%) 
Deviation from baseline 



All early-implementing schools 

8 th grade attendance rate (%) 86.9 87.6 87.4 85.5 87.7 85.5 85.8 85.3 85.7 88.1 

Deviation from baseline 3.0*** 3.8*** 3.5*** 1.9 2.0 1.4** 1.7*** 1.2** 2.0*** 3.0*** 
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The Talent Development Evaluation 
TR Table A.7 (continued) 

Year-by-Year Levels and Impacts for Attendance Rate 

for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

8 th grade attendance rate (%) 
Deviation from baseline 


- 1.6 


0.0 


0.9 


1.6 


0.4 


-0.05 


0.00 


0.03 


0.05 


0.01 


School B 

8 th grade attendance rate (%) 
Deviation from baseline 


- 0.6 


1.8 


1.2 


- 0.1 


-2.4 


- 0.02 


0.06 


0.04 


0.00 


-0.08 


School C 

8 th grade attendance rate (%) 
Deviation from baseline 


2.7 


2.2 


2.7 


1.2 




0.09 


0.07 


0.09 


0.04 




School D 

8 th grade attendance rate (%) 
Deviation from baseline 


5.3 


0.9 


-0.5 


-2.9 




0.18 


0.03 


- 0.02 


- 0.10 




School E 

8 th grade attendance rate (%) 
Deviation from baseline 


2.4 


5.5 ** 


7 3 *** 






0.08 


0.18 


** 0.24 *** 






School F 

8 th grade attendance rate (%) 
Deviation from baseline 
























All early-implementing schools 
8 th grade attendance rate (%) 
Deviation from baseline 


1.6 


2.1 * 


2.3 * 


- 0.1 


- 1.0 


0.05 


0.07 


0.08 


0.00 


-0.03 
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TR Table A.7 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4. and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a given 
school year. 
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The Talent Development Evaluation 
TR Table A.8 



Year-by-Year Levels and Impacts for Attendance Rates Greater Than or Equal to 90 Percent 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 

School Cluster Year 1 Year 2 Year 3 Year 4 Year 5 Year 1 Year 2 Year 3 Year 4 Year 5 

School A 



Attendance rate of 90% or higher (%) 


48.4 


48.2 


52.7 


55.7 


58.7 


54.4 


54.1 


52.0 


52.7 


54.3 


Deviation from baseline 


0.0 


- 0.2 


4.3 


7.3 


10.3 


7.3 ** 


7 0 ** 


5.0 * 


5.6 ** 


7 2** 


School B 






















Attendance rate of 90% or higher (%) 


62.5 


69.3 


61.9 


61.7 


64.1 


58.1 


55.8 


54.2 


55.5 


61.5 


Deviation from baseline 


OO 
O O 


15.6 


8.3 


8.0 


10.4 


6.1 


3.9 


2.3 


3.5 


9.6 


School C 






















Attendance rate of 90% or higher (%) 


39.6 


38.4 


47.8 


50.5 




51.0 


53.5 


46.3 


49.7 




Deviation from baseline 


9.7 


8.5 


17.9 ** 


20.6 ** 




5.7 


8.2 ** 


0.9 


4.4 




School D 






















Attendance rate of 90% or higher (%) 


70.7 


56.6 


51.9 


43.7 




36.3 


41.5 


42.8 


46.0 




Deviation from baseline 


99 9 *** 


oo 

OO 


4.1 


-4.1 




-7.5 


-2.3 


- 1.0 


2.2 




School E 






















Attendance rate of 90% or higher (%) 


54.5 


59.7 


74.6 






49.5 


50.7 


50.2 






Deviation from baseline 


0.0 


5.3 


20.1 *** 






-1.9 


-0.7 


- 1.2 







School F 

Attendance rate of 90% or higher (%) 
Deviation from baseline 





All early-implementing schools 
Attendance rate of 90% or higher (%) 
Deviation from baseline 


55.1 
8.3 ** 


54.5 
7.6 ** 


57.8 

10 9 *** 


52.9 
8.0 * 


61.4 

10.4 


49.8 

2.0 


51.1 

3.2 * 


49.1 

1.2 


51.0 

3.9 * 


57.9 
8.4 ** 



(continued) 
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TR Table A.8 (continued) 

Year-by-Year Levels and Impacts for Attendance Rates Greater Than or Equal to 90 Percent 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


-7.3 


-7.2 


-0.7 


1.6 


3.1 


-0.15 


-0.15 


- 0.01 


0.03 


0.06 


School B 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


2.6 


11.8 


5.9 


4.4 


0.9 


0.05 


0.24 


0.12 


0.09 


0.02 


School C 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


4.0 


0.3 


17.0 * 


16.3 * 




0.08 


0.01 


0.34 * 


0.33 * 




School D 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


30.4 *** 


11.2 


5.1 


-6.3 




0.62 *** 


0.23 


0.10 


-0.13 




School E 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


1.9 


5.9 


2 i 3 *** 






0.04 


0.12 


0.43 *** 






School F 

Attendance rate of 90% or higher (%) 
Deviation from baseline 













































All early-implementing schools 
Attendance rate of 90% or higher (%) 
Deviation from baseline 


6.3 


4.4 


9 7 ** 


4.0 


2.0 


0.13 


0.09 


0.20 ** 


0.08 


0.04 



(continued) 
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TR Table A.8 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3. Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or 
data were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a given 
school year. 
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The Talent Development Evaluation 
TR Table A.9 



Year-by-Year Levels and Impacts for Attendance Rates Less Than or Equal to 80 Percent 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 



School Cluster 




Talent Development Schools 






Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















Attendance rate of 80% or lower (%) 


24.4 


22.2 


25.4 


22.5 


19.9 


21.3 


22.0 


23.4 


23.1 


22.5 


Deviation from baseline 


-1.4 


-3.6 


-0.3 


-3.3 


-5.8 


7 *** 


-6.0 *** 


-4.6 ** 


_4 9 ** 


-5.5 *** 


School B 






















Attendance rate of 80% or lower (%) 


16.2 


11.1 


14.8 


15.8 


15.4 


18.4 


21.1 


22.9 


20.9 


18.6 


Deviation from baseline 


-0.8 


-5.9 


-2.2 


-1.2 


-1.6 


-7.1 * 


-4.5 


-2.7 


-4.6 


-7.0 * 


School C 






















Attendance rate of 80% or lower (%) 


35.8 


34.0 


29.3 


26.0 




25.2 


24.0 


25.9 


26.7 




Deviation from baseline 


-9.6 


-11.4 


-16.0 ** 


-19.4 ** 




-5.1 * 


-6.3 ** 


-4.4 


-3.6 




School D 






















Attendance rate of 80% or lower (%) 


11.8 


22.9 


22.6 


31.7 




35.4 


29.0 


30.9 


29.5 




Deviation from baseline 


-17.0 ** 


-5.9 


-6.2 


2.9 




3.5 


-2.9 


-1.0 


-2.4 




School E 






















Attendance rate of 80% or lower (%) 


20.8 


13.9 


10.3 






24.8 


24.7 


24.9 






Deviation from baseline 


-4.7 


-11.6 ** 


-15.2 *** 






0.9 


0.8 


1.0 







School F 

Attendance rate of 80% or lower (%) 
Deviation from baseline 



All early-implementing schools 

Attendance rate of 80% or lower (%) 21.8 20.8 20.5 24.0 17.7 25.0 24.2 25.6 25.0 20.6 

Deviation from baseline -6.7** -7 .7 *** -8.0*** -5.2* -3.7 -2.9 ** -3.8*** -2.3* -3.9** -6.2*** 



(continued) 
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The Talent Development Evaluation 
TR Table A.9 (continued) 



Year-by-Year Levels and Impacts for Attendance Rates Less Than or Equal to 80 Percent 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 








Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


5.4 


2.4 


4.3 


1.7 


-0.4 


0.11 


0.05 


0.09 


0.03 


-0.01 


School B 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


6.3 


-1.4 


0.4 


3.5 


5.3 


0.13 


-0.03 


0.01 


0.07 


0.11 


School C 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


-4.5 


-5.1 


-11.7 


-15.8 * 




-0.09 


-0.11 


-0.24 


-0.33 * 




School D 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


-20.4 ** 


-3.0 


-5.3 


5.3 




-0.43 ** 


-0.06 


-0.11 


0.11 




School E 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


-5.6 


-12.4 ** 


-16.2 *** 






-0.12 


-0.26 ** 


-0.34 *** 






School F 

Attendance rate of 80% or lower (%) 
Deviation from baseline 
























All early-implementing schools 
Attendance rate of 80% or lower (%) 






















Deviation from baseline 


-3.8 


-3.9 


-5.7 * 


-1.3 


2.5 


-0.08 


-0.08 


-0.12 * 


-0.03 


0.05 



(continued) 
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TR Table A.9 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4. and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 11 Talent Development schools and the 
18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a given 
school year. 
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The Talent Development Evaluation 
TR Table A.10 



Year-by-Year Levels and Impacts for One- Year Promotion Rate 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 



School A 

Promoted to 
Deviation 



School B 

Promoted to 
Deviation 

School C 

Promoted to 
Deviation 

School D 

Promoted to 
Deviation 

School E 

Promoted to 
Deviation 

School F 

Promoted to 
Deviation 



Outcome Levels Compared with Baseline Average 





Talent Development Schools 






Non-Talent Development Schools 




Year 1 


Year 2 Year 3 Year 4 


Year 5 


Year 1 


Year 2 Year 3 Year 4 


Year 5 



9th grade 


99.5 


98.9 


100.1 


99.8 


99.0 


98.0 


97.4 


98.9 


96.4 


98.4 


from baseline 


1.5 


0.9 


2.1 


1.8 


1.0 


-0.1 


-0.8 


0.8 


_) 7 ** 


0.2 


9th grade 


99.5 


98.2 


99.8 


98.2 


99.1 


99.0 


96.2 


98.2 


97.5 


98.6 


from baseline 


0.6 


-0.6 


1.0 


-0.6 


0.3 


0.7 


-2.0 ** 


-0.1 


-0.8 


0.3 


9th grade 


94.4 


96.9 


100.4 


100.4 




98.3 


98.8 


97.5 


96.8 




from baseline 


-2.1 


0.4 


3.9 


3.9 




-0.2 


0.3 


-0.9 


-1.7 * 




9th grade 


96.2 


98.7 


99.5 


97.9 




95.5 


98.3 


95.6 


93.4 




from baseline 


-1.3 


1.2 


2.0 


0.4 




-0.8 


2.0 


-0.7 


-2.9 




9th grade 


98.0 


97.5 








99.1 


96.0 








from baseline 


-0.1 


-0.6 








1.1 


-2.0 ** 








9th grade 


98.8 


97.8 








99.1 


97.3 








from baseline 


3.6 


2.7 








1.3 * 


-0.5 









All early-implementing schools 

Promoted to 9th grade 97.7 98.0 100.0 99.1 99.1 98.2 97.3 97.6 96.0 98.5 

Deviation from baseline 0.4 0.7 2.2 *** 1.4 0.6 0.3 -0.5 -0.2 -1.8 *** 0.2 



(continued) 
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The Talent Development Evaluation 
TR Table A.10 (continued) 

Year-by-Year Levels and Impacts for One- Year Promotion Rate 
for Eighth-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 






School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

Promoted to 9th grade 
Deviation from baseline 


1.6 


1.6 


1.3 


3.6 


0.8 


0.07 


0.07 


0.06 


0.15 


0.03 


School B 

Promoted to 9th grade 
Deviation from baseline 


-0.1 


1.4 


1.1 


0.2 


0.0 


0.00 


0.06 


0.04 


0.01 


0.00 


School C 

Promoted to 9th grade 
Deviation from baseline 


-1.9 


0.0 


4.8 * 


5.6 ** 




-0.08 


0.00 


0.20 * 


0.24 ** 




School D 

Promoted to 9th grade 
Deviation from baseline 


-0.5 


-0.9 


2.7 


3.2 




-0.02 


-0.04 


0.12 


0.14 




School E 

Promoted to 9th grade 
Deviation from baseline 


-1.2 


1.4 








-0.05 


0.06 








School F 

Promoted to 9th grade 
Deviation from baseline 


2.3 


3.2 








0.10 


0.13 































All early-implementing schools 
Promoted to 9th grade 
Deviation from baseline 


0.0 


1.1 


2.5 *** 


3.1 ** 


0.4 


0.00 


0.05 


o.n *** 


0.13 ** 


0.02 
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TR Table A.10 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 
3, Year 4, and Year 5 averages, respectively. 

The impacts for Year 1 , Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a 
prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 
10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across school averages at the bottom 
of each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 

Eighth-grade students were considered promoted if they were listed as 9th-grade students in the district’s administrative data file one year after the current year. Students 
whose records were not included on the data file one year after the current year, for whatever reason, were not in the analysis sample for this outcome. 
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The Talent Development Evaluation 
TR Table B.l 

Year-by-Year Levels and Impacts for SAT-9 Math Total NCE Scores 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 



School Cluster 




Talent Development Schools 






Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















7th-grade average NCE 


34.5 


35.5 


34.7 


37.0 


38.1 


37.7 


37.5 


34.7 


32.8 


37.2 


Deviation from baseline 


2.3 


3.3 


2.5 


4.7 


5.8 


4 2 *** 


4 q *** 


1.2 


-0.7 


3 7 *** 


School B 






















7th-grade average NCE 


39.2 


39.6 


35.6 


37.9 


40.9 


40.1 


38.7 


39.6 


34.1 


39.0 


Deviation from baseline 


0.0 


0.4 


-3.6 


-1.3 


1.7 


4.3 * 


2.8 


3.7 * 


-1.7 


3.2 


School C 






















7th-grade average NCE 


31.3 


26.2 


29.7 


32.7 




31.9 


29.3 


31.0 


33.0 




Deviation from baseline 


1.9 


-3.2 


0.3 


3.3 




1.2 


-1.4 


0.3 


2.3 * 




School D 






















7th-grade average NCE 


32.4 


30.1 


31.5 


36.3 




35.0 


33.4 


31.1 


35.9 




Deviation from baseline 


0.9 


-1.4 


- 0.1 


* 

* 

oo 




to 

bo 

* 

* 


1.2 


- 1.0 


* 

* 

OO 

CO 




School E 






















7th-grade average NCE 


33.4 


34.1 


36.9 






31.8 


31.6 


36.0 






Deviation from baseline 


-5.5 * 


-4.8 


- 2.1 






-3 3 *** 


-3.5 *** 


0.9 






School F 






















7th-grade average NCE 


35.4 


36.3 








29.8 


31.0 








Deviation from baseline 


3.0 


3.9 








-2.8 *** 


- 1.6 ** 









All early-implementing schools 

7th-grade average NCE 34.4 33.7 33.7 36.0 39.5 34.4 33.6 34.5 33.9 38.1 

Deviation from baseline 0.4 -0.3 -0.6 2.9 * 3.8 1.1 * 0.2 1.0 0.9 3.5 *** 



(continued) 
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TR Table B.l (continued) 

Year-by-Year Levels and Impacts for SAT-9 Math Total NCE Scores 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

7th-grade average NCE 
Deviation from baseline 


-1.9 


-0.7 


1.3 


5.5 


2.1 


-0.13 


-0.05 


0.09 0.38 


0.14 


School B 

7th-grade average NCE 
Deviation from baseline 


-4.3 


-2.4 


-7.3 * 


0.4 


-1.5 


-0.29 


-0.16 


-0.50 * 0.03 


-0.10 


School C 

7th-grade average NCE 
Deviation from baseline 


0.6 


-1.8 


0.0 


1.0 




0.04 


-0.13 


0.00 0.07 




School D 

7th-grade average NCE 
Deviation from baseline 


-2.0 


-2.6 


1.0 


1.0 




-0.13 


-0.18 


0.07 0.07 




School E 

7th-grade average NCE 
Deviation from baseline 


-2.2 


-1.2 


-2.9 






-0.15 


-0.09 


-0.20 




School F 

7th-grade average NCE 
Deviation from baseline 


5.9 ** 


5.5 * 








0.40 ** 


0.38 


* 






All early-implementing schools 
7th-grade average NCE 
Deviation from baseline 


-0.6 


-0.5 


-1.6 


2.0 


0.3 


-0.04 


-0.04 


-0.11 0.14 


0.02 



(continued) 
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TR Table B.l (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes students 
not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 1 45 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and the 
18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school estimates, 
which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of each page 
include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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TR Table B.2 

Year-by-Year Levels and Impacts for SAT-9 Math Total Scores At or Above Grade Level 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 



School Cluster 




Talent Development Schools 




Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















At or above grade level (%) 


20.0 


15.1 


17.3 


15.2 


16.9 


21.2 


20.2 


15.2 


9.8 


18.5 


Deviation from baseline 


7.5 


2.7 


4.8 


2.7 


4.4 


7 g *** 


6.8 *** 


1.9 


-3.6 


5.1 ** 


School B 






















At or above grade level (%) 


26.4 


25.0 


14.4 


17.3 


21.8 


26.2 


23.3 


26.1 


12.4 


21.4 


Deviation from baseline 


1.1 


-0.3 


-10.9 


-7.9 


-3.4 


9.6 ** 


6.7 


9.5 ** 


-4.3 


4.8 


School C 






















At or above grade level (%) 


12.9 


7.8 


5.1 


6.8 




12.0 


6.6 


6.6 


7.7 




Deviation from baseline 


6.3 


1.2 


-1.5 


0.2 




3.9 ** 


-1.5 


-1.5 


-0.4 




School D 






















At or above grade level (%) 


8.5 


3.5 


7.1 


13.7 




12.9 


14.2 


4.8 


12.4 




Deviation from baseline 


-0.9 


C/i 

oo 

* 


- 2.2 


4.4 




3.6 


4 9 ** 


-4.5 ** 


3.2 




School E 






















At or above grade level (%) 


13.9 


9.4 


15.2 






9.6 


7.4 


15.8 






Deviation from baseline 


- 6.1 


- 10.6 


-4.8 






- 6.6 ** 


_g 7 *** 


-0.4 






School F 






















At or above grade level (%) 


14.5 


11.8 








6.3 


6.5 








Deviation from baseline 


3.3 


0.5 








-5.2 *** 


-5.0 *** 









All early-implementing schools 

At or above grade level (%) 16.0 12.1 11.8 13.3 19.4 14.7 13.0 13.7 10.6 19.9 

Deviation from baseline 1.9 -2.0 -2.9 -0.2 0.5 2.2 ** 0.5 1.0 -1.3 5.0 ** 



(continued) 
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TR Table B.2 (continued) 



Year-by-Year Levels and Impacts for SAT-9 Math Total Scores At or Above Grade Level 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

At or above grade level (%) 
Deviation from baseline 


-0.4 


-4.2 


3.0 


6.3 


-0.7 


-0.01 


-0.13 


0.09 0.19 


-0.02 


School B 

At or above grade level (%) 
Deviation from baseline 


-8.5 


-7.0 


-20.3 ** 


-3.7 


-8.2 


-0.26 


-0.22 


-0.63** -0.11 


-0.25 


School C 

At or above grade level (%) 
Deviation from baseline 


2.4 


2.7 


0.0 


0.6 




0.07 


0.08 


0.00 0.02 




School D 

At or above grade level (%) 
Deviation from baseline 


-4.4 


-10.6 ** 


2.3 


1.3 




-0.14 


-0.33 


** 0.07 0.04 




School E 

At or above grade level (%) 
Deviation from baseline 


0.5 


-1.9 


-4.4 






0.01 


-0.06 


-0.14 




School F 

At or above grade level (%) 
Deviation from baseline 


8.5 * 


5.5 








0.26 * 


0.17 








All early-implementing schools 
At or above grade level (%) 
Deviation from baseline 


-0.3 


-2.6 


-3.9 


1.1 


-4.5 


-0.01 


-0.08 


-0.12 0.03 


-0.14 
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TR Table B.2 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2o and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1. Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 11 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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TR Table B.3 



Year-by-Year Levels and Impacts for SAT-9 Math Total Scores in the Bottom Quartile 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 




Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















In the bottom quartile (%) 


54.1 


56.9 


58.9 


54.8 


49.5 


50.1 


51.4 


57.3 


65.3 


52.5 


Deviation from baseline 


- 12.0 


-9.2 


-7.2 


-11.3 


-16.6 


- 10.0 *** 


* 

* 

OO 

OO 


-2.9 


5.1 


-7.6 ** 


School B 






















In the bottom quartile (%) 


44.0 


46.3 


53.3 


46.4 


39.5 


43.5 


47.7 


42.9 


62.0 


45.6 


Deviation from baseline 


-5.4 


-3.1 


3.9 


-3.0 


-9.9 


-10.3 


- 6.1 


- 10.8 


8.2 


- 8.2 


School C 






















In the bottom quartile (%) 


62.5 


80.7 


75.4 


63.5 




65.2 


73.9 


70.5 


67.6 




Deviation from baseline 


- 11.8 


6.5 


1.2 


- 10.8 




-4.9 


3.8 


0.3 


-2.5 




School D 






















In the bottom quartile (%) 


66.0 


71.9 


65.4 


54.7 




56.6 


62.4 


71.7 


57.6 




Deviation from baseline 


-2.5 


3.4 


-3.1 


-13.8 * 




- 10.1 * 


-4.2 


5.0 


-9.0 




School E 






















In the bottom quartile (%) 


60.9 


59.9 


58.5 






65.8 


68.6 


56.6 






Deviation from baseline 


13.9 


12.8 


11.5 






OO 

* 

* 


H 2 *** 


- 0.8 






School F 






















In the bottom quartile (%) 


56.7 


53.6 








72.7 


71.0 








Deviation from baseline 


-8.4 


-11.4 








8.5 *** 


6.8 ** 










All early-implementing schools 






















In the bottom quartile (%) 


57.4 


61.5 


62.3 


54.8 


44.5 


59.0 


62.5 


59.8 


63.1 


49.0 


Deviation from baseline 


-4.3 


- 0.2 


1.3 


_9 7 ** 


-13.2 * 


-3.1 * 


0.5 


- 1.8 


0.4 


_7 9 ** 



(continued) 
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Year-by-Year Levels and Impacts for SAT-9 Math Total Scores in the Bottom Quartile 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

In the bottom quartile (%) 
Deviation from baseline 


-1.9 


-0.4 


-4.3 


-16.3 


-9.0 


-0.04 


-0.01 


-0.09 -0.34 


-0.19 


School B 

In the bottom quartile (%) 
Deviation from baseline 


4.9 


3.0 


14.7 


-11.2 


-1.7 


0.10 


0.06 


0.31 -0.24 


-0.04 


School C 

In the bottom quartile (%) 
Deviation from baseline 


-6.8 


2.7 


0.8 


-8.2 




-0.14 


0.06 


0.02 -0.17 




School D 

In the bottom quartile (%) 
Deviation from baseline 


7.6 


7.7 


-8.1 


-4.7 




0.16 


0.16 


-0.17 -0.10 




School E 

In the bottom quartile (%) 
Deviation from baseline 


5.4 


1.6 


12.3 






0.11 


0.03 


0.26 




School F 

In the bottom quartile (%) 
Deviation from baseline 


-16.8 * 


-18.2 * 








-0.35 * 


-0.38 * 








All early-implementing schools 
In the bottom quartile (%) 
Deviation from baseline 


-1.3 


-0.6 


3.1 


-10.1 * 


-5.3 


-0.03 


-0.01 


0.06 -0.21 * 


-0.11 



(continued) 
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TR Table B.3 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving NCE Scores 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 




Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















7th-grade average NCE 


36.4 


38.4 


36.5 


38.7 


40.1 


37.3 


37.7 


35.2 


33.2 


37.8 


Deviation from baseline 


4.4 


6.4 * 


4.5 


6.7 * 


8.1 ** 


4 i *** 


4.5 *** 


2.0 * 


0.0 


4 6 *** 


School B 






















7th-grade average NCE 


39.2 


41.3 


36.9 


39.3 


42.6 


38.7 


38.8 


39.7 


35.0 


39.4 


Deviation from baseline 


-0.4 


1.8 


- 2.6 


-0.3 


3.0 


4.2 ** 


4.3 ** 


5.3 ** 


0.6 


5.0 ** 


School C 






















7th-grade average NCE 


31.9 


25.9 


30.1 


34.0 




32.6 


29.7 


31.4 


33.2 




Deviation from baseline 


2.4 


-3.6 


0.6 


4.5 




2.2 


-0.7 


1.0 


2.8 ** 




School D 






















7th-grade average NCE 


32.9 


32.0 


32.0 


37.3 




35.8 


34.7 


33.1 


33.9 




Deviation from baseline 


1.7 


0.8 


0.8 


6.1 ** 




3.3 * 


2.2 


0.6 


1.4 




School E 






















7th-grade average NCE 


35.5 


33.9 


38.2 






32.3 


31.9 


36.6 






Deviation from baseline 


-3.0 


-4.6 


-0.3 






_2 9 ** 


-3 3 *** 


1.4 






School F 






















7th-grade average NCE 


36.4 


35.1 








29.9 


31.3 








Deviation from baseline 


3.2 


2.0 








- 2.8 *** 


-1.4 










All early-implementing schools 






















7th-grade average NCE 


35.4 


34.4 


34.8 


37.3 


41.3 


34.4 


34.0 


35.2 


33.8 


38.6 


Deviation from baseline 


1.4 


0.4 


0.6 


4.2 *** 


5.6 ** 


1.4 ** 


0.9 * 


2.1 *** 


1.2 


4 g *** 



(continued) 
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Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving NCE Scores 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 



Impact 

Year 1 Year 2 Year 3 Year 4 Year 5 



Impact Effect Size 

Year 1 Year 2 Year 3 Year 4 Year 5 



School A 

7th-grade average NCE 
Deviation from baseline 

School B 

7th-grade average NCE 
Deviation from baseline 

School C 

7th-grade average NCE 
Deviation from baseline 

School D 

7th-grade average NCE 
Deviation from baseline 

School E 

7th-grade average NCE 
Deviation from baseline 

School F 

7th-grade average NCE 
Deviation from baseline 



0.3 



1.8 2.5 6.7 * 3.5 



-4.6 -2.6 -7.9 ** -0.9 -1.9 



0.2 -2.9 -0.4 1.7 



-1.6 -1.4 0.2 



4.6 



-0.1 -1.3 -1.7 



6.0 * 3.4 



0.02 0.12 0.16 0.44 * 0.23 



-0.31 -0.17 -0.52 ** -0.06 -0.13 



0.01 -0.19 -0.02 0.11 



-0.11 -0.09 0.01 0.31 



-0.01 -0.09 -0.11 



0.40 * 0.22 



All early-implementing schools 
7th-grade average NCE 

Deviation from baseline 0.0 -0.5 -1.5 3.1 * 0.8 0.00 -0.03 -0.10 0.20 * 0.05 



(continued) 
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TR Table B.4 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1. Year 2, Year 3, Year 4. and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving Scores At or Above Grade Level 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 

School A 

At or above grade level (%) 
Deviation from baseline 

School B 

At or above grade level (%) 
Deviation from baseline 

School C 

At or above grade level (%) 
Deviation from baseline 

School D 

At or above grade level (%) 
Deviation from baseline 

School E 

At or above grade level (%) 
Deviation from baseline 

School F 

At or above grade level (%) 
Deviation from baseline 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


21.4 


20.2 


18.5 


16.5 


20.3 


19.2 


20.3 


16.2 


10.8 


20.4 


11.7 


10.4 


OO 
O O 


6.8 


10.6 


6.2 ** 


7 3 *** 


3.2 


- 2.2 


7 4 *** 


28.4 


29.2 


19.4 


18.7 


26.8 


21.8 


23.7 


26.5 


15.0 


23.3 


2.3 


3.1 


-6.7 


-7.4 


0.7 


7.2 * 


9 2 ** 


l p9 *** 


0.5 


OO 

bo 

* 

* 


14.0 


7.1 


6.0 


8.7 




12.8 


6.9 


7.4 


10.2 




6.0 


-0.9 


- 2.0 


0.7 




5.1 ** 


-0.9 


-0.4 


2.4 




10.6 


5.3 


7.3 


15.7 




14.8 


15.5 


10.2 


11.3 




0.3 


-5.0 


-3.0 


5.4 




3.9 


4.6 * 


-0.7 


0.4 




13.5 


10.4 


17.9 






10.7 


7.9 


18.0 






-7.0 


- 10.1 


- 2.6 






-5.3 ** 


_g i *** 


2.0 






15.9 


12.7 








6.4 


7.4 








3.4 


0.3 








_g 2 *** 


-4.3 *** 









All early-implementing schools 

At or above grade level (%) 17.3 14.1 13.8 14.9 23.5 14.3 13.6 15.7 11.8 21.8 

Deviation from baseline 2.8 -0.4 -1.1 1.4 5.7 2.0 * 1.3 3.2 *** 0.3 8.1 *** 



(continued) 
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Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving Scores At or Above Grade Level 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

At or above grade level (%) 
Deviation from baseline 


5.5 


3.1 


5.5 


9.0 


3.2 


0.17 


0.10 


0.17 


0.28 


0.10 


School B 

At or above grade level (%) 
Deviation from baseline 


-4.9 


-6.1 


-18.6 ** 


-7.9 


-8.1 


-0.15 


-0.19 


-0.58 ** 


-0.25 


-0.25 


School C 

At or above grade level (%) 
Deviation from baseline 


0.9 


0.0 


-1.6 


-1.8 




0.03 


0.00 


-0.05 


-0.06 




School D 

At or above grade level (%) 
Deviation from baseline 


-3.6 


-9.6 ** 


-2.3 


5.1 




-0.11 


-0.30 ** 


-0.07 


0.16 




School E 

At or above grade level (%) 
Deviation from baseline 


-1.7 


-2.0 


-4.7 






-0.05 


-0.06 


-0.15 






School F 

At or above grade level (%) 
Deviation from baseline 


8.6 


4.5 








0.27 


0.14 










All early-implementing schools 
At or above grade level (%) 
Deviation from baseline 


0.8 


-1.7 


-4.3 


1.1 


-2.4 


0.03 


-0.05 


-0.14 


0.03 


-0.08 
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TR Table B.5 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4. and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4. and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
was not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table B.6 

Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving Scores in the Bottom Quartile 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 

Outcome Levels Compared with Baseline Average 



Talent Development Schools Non-Talent Development Schools 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















In the bottom quartile (%) 


52.6 


44.9 


55.7 


42.5 


35.4 


52.9 


51.7 


56.2 


61.4 


50.4 


Deviation from baseline 


-13.6 


-21.3 ** 


-10.5 


-23.6 ** 


-30.8 *** 


-9.2 *** 


-10.3 *** 


L/1 

bo 

* 


-0.7 


-11.6 *** 


School B 






















In the bottom quartile (%) 


43.9 


45.4 


52.7 


39.3 


34.1 


48.6 


49.2 


44.3 


56.7 


45.7 


Deviation from baseline 


-5.2 


-3.7 


3.6 


-9.7 


-15.0 


-10.3 * 


-9.7 * 


-14.6 ** 


-2.2 


-13.2 ** 


School C 






















In the bottom quartile (%) 


62.7 


83.0 


72.7 


63.0 




64.5 


72.8 


65.5 


62.7 




Deviation from baseline 


-10.6 


9.7 


-0.6 


-10.3 




-6.4 * 


1.8 


-5.4 


oo 

to 

* 

* 




School D 






















In the bottom quartile (%) 


63.3 


65.8 


62.3 


54.0 




53.9 


57.6 


62.3 


60.8 




Deviation from baseline 


-5.8 


-3.3 


-6.8 


-15.1 * 




-10.3 * 


-6.6 


-1.9 


-3.4 




School E 






















In the bottom quartile (%) 


56.9 


60.0 


49.6 






63.9 


64.9 


54.3 






Deviation from baseline 


9.6 


12.7 


2.3 






5.6 * 


6.6 ** 


-4.0 






School F 






















In the bottom quartile (%) 


57.9 


59.8 








71.7 


66.4 








Deviation from baseline 


-4.6 


-2.7 








6.9 ** 


1.6 










All early-implementing schools 






















In the bottom quartile (%) 


56.2 


59.8 


58.6 


49.7 


34.8 


59.3 


60.4 


56.5 


60.4 


48.1 


Deviation from baseline 


-5.0 


-1.4 


-2.4 


-14.7 *** 


-22.9 *** 


-3.9 ** 


to 

oo 

* 


_6.4 *** 


-3.6 


-12.4 *** 
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The Talent Development Evaluation 
TR Table B.6 (continued) 



Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving Scores in the Bottom Quartile 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

In the bottom quartile (%) 
Deviation from baseline 


-4.4 


-10.9 


-4.7 


-22.9 ** 


-19.1 * 


-0.09 


-0.23 


-0.10 -0.48 ** 


-0.40 * 


School B 

In the bottom quartile (%) 
Deviation from baseline 


5.0 


6.0 


18.2 


-7.5 


-1.8 


0.11 


0.13 


0.38 -0.16 


-0.04 


School C 

In the bottom quartile (%) 
Deviation from baseline 


-4.2 


7.9 


4.8 


-2.1 




-0.09 


0.17 


0.10 -0.04 




School D 

In the bottom quartile (%) 
Deviation from baseline 


4.5 


3.3 


-4.8 


-11.8 




0.09 


0.07 


-0.10 -0.25 




School E 

In the bottom quartile (%) 
Deviation from baseline 


4.1 


6.2 


6.3 






0.09 


0.13 


0.13 




School F 

In the bottom quartile (%) 
Deviation from baseline 


-11.6 


-4.3 








-0.24 


-0.09 








All early-implementing schools 
In the bottom quartile (%) 
Deviation from baseline 


-1.1 


1.3 


4.0 


-H I ** 


-10.5 


-0.02 


0.03 


0.08 -0.23 ** 


-0.22 



(continued) 
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TR Table B.6 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4 and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4 and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4 and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all eighth-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table B.7 



Year-by-Year Levels and Impacts for SAT-9 Reading NCE Scores 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 

Year 1 Year 2 Year 3 Year 4 Year 5 Year 1 Year 2 Year 3 Year 4 Year 5 



School A 
7th-grade average 
Deviation from 

School B 
7th-grade average 
Deviation from 

School C 
7th-grade average 
Deviation from 

School D 
7th-grade average 
Deviation from 

School E 
7th-grade average 
Deviation from 

School F 
7th-grade average 
Deviation from 



NCE 


36.9 


37.0 


41.7 


38.3 


39.5 


39.0 


38.8 


36.4 


34.7 


40.0 


baseline 


-0.8 


-0.7 


4.1 


0.7 


1.8 


2.2 ** 


2 0** 


-0.4 


-2.1 ** 


3.2 


NCE 


40.5 


43.2 


41.6 


41.0 


45.2 


40.4 


41.3 


41.4 


37.7 


43.0 


baseline 


0.5 


3.1 


1.6 


0.9 


5.1 


1.0 


1.9 


2.0 


-1.7 


3.5 


NCE 


35.5 


30.4 


33.6 


34.6 




35.7 


32.2 


33.9 


35.8 




baseline 


4.2 


-1.0 


2.2 


3.2 




2 7** 


-0.8 


0.9 


2.8 ** 




NCE 


33.4 


32.3 


31.7 


36.3 




35.5 


35.4 


30.4 


35.4 




baseline 


-1.5 


-2.7 


-3.3 


1.3 




0.6 


0.5 


-4.4 * 


0.6 




NCE 


36.5 


36.1 


39.6 






34.3 


33.4 


38.8 






baseline 


-2.9 


-3.3 


0.2 






_9 g *** 


7 *** 


1.7 * 






NCE 


34.4 


32.4 








32.3 


33.3 








baseline 


1.4 


-0.6 








_9 g *** 


* 

* 

OO 












* 



All early-implementing schools 

7th-grade average NCE 36.2 35.2 37.6 37.5 42.3 36.2 35.7 36.2 35.9 41.5 

Deviation from baseline 0.1 -0.8 1.0 1.5 3.5 0.2 -0.3 -0.1 -0.1 3.3 *** 



(continued) 
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The Talent Development Evaluation 
TR Table B.7 (continued) 

Year-by-Year Levels and Impacts for SAT-9 Reading NCE Scores 
for Seventh-Grade Students in Early -Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 



School A 

7th-grade average NCE 
Deviation from baseline 

School B 

7th-grade average NCE 
Deviation from baseline 

School C 

7th-grade average NCE 
Deviation from baseline 

School D 

7th-grade average NCE 
Deviation from baseline 

School E 

7th-grade average NCE 
Deviation from baseline 

School F 

7th-grade average NCE 
Deviation from baseline 



Impact 

Year 1 Year 2 Year 3 Year 4 Year 5 



-3.0 -2.7 4.5 



2.8 -1.3 



-0.5 



1.2 -0.4 



2.6 1.6 



1.4 -0.1 



1.4 0.4 



-2.2 -3.2 1.1 0.7 

-0.1 0.4 -1.5 

4.2 1.2 



Impact Effect Size 

Year 1 Year 2 Year 3 Year 4 Year 5 



-0.17 

-0.03 

0.08 

-0.13 



-0.16 

0.07 

- 0.01 

-0.19 



0.26 

- 0.02 

0.08 

0.07 



0.16 -0.08 

0.15 0.09 

0.02 
0.04 



-0.01 0.02 -0.09 

0.25 0.07 



All early-implementing schools 
7th-grade average NCE 

Deviation from baseline 0.0 -0.5 1.0 1.6 0.1 0.00 -0.03 0.06 0.09 0.01 



(continued) 
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TR Table B.7 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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TR Table B.8 

Year-by-Year Levels and Impacts for SAT-9 Reading Scores At or Above Grade Level 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 

Outcome Levels Compared with Baseline Average 



Talent Development Schools Non-Talent Development Schools 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















At or above grade level (%) 


24.0 


22.2 


34.1 


25.3 


27.0 


24.4 


25.8 


22.8 


18.9 


24.7 


Deviation from baseline 


-0.4 


- 2.2 


9.7 


0.9 


2.6 


0.3 


1.7 


- 1.2 


-5.2 ** 


0.6 


School B 






















At or above grade level (%) 


31.5 


36.3 


31.5 


27.7 


34.4 


25.6 


30.4 


34.3 


25.6 


30.4 


Deviation from baseline 


2.6 


7.4 


2.6 


- 1.2 


5.5 


-3.1 


1.7 


5.7 * 


-3.0 


1.8 


School C 






















At or above grade level (%) 


24.0 


10.0 


14.1 


13.9 




20.3 


13.7 


17.8 


15.8 




Deviation from baseline 


9.5 * 


-4.5 


-0.4 


- 0.6 




3.8 ** 


- 2.8 


1.3 


-0.7 




School D 






















At or above grade level (%) 


16.1 


14.9 


11.2 


19.7 




17.8 


20.3 


13.3 


16.7 




Deviation from baseline 


- 1.1 


- 2.2 


-5.9 


2.5 




-0.9 


1.6 


-5.4 


- 2.1 




School E 






















At or above grade level (%) 


22.7 


19.9 


23.9 






17.9 


16.0 


22.3 






Deviation from baseline 


-5.7 


-8.5 


-4.5 






-5.4 *** 


j~! 4 *** 


- 1.1 






School F 






















At or above grade level (%) 


14.7 


11.2 








13.7 


16.1 








Deviation from baseline 


- 2.6 


- 6.2 








_5 7 *** 


-3.3 ** 










All early-implementing schools 






















At or above grade level (%) 


22.2 


19.1 


23.0 


21.6 


30.7 


20.0 


20.4 


22.1 


19.2 


27.6 


Deviation from baseline 


0.4 


-2.7 


0.3 


0.4 


4.0 


- 1.8 * 


-1.4 


- 0.2 


to 

bo 

* 


1.2 
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The Talent Development Evaluation 
TR Table B.8 (continued) 

Year-by-Year Levels and Impacts for SAT-9 Reading Scores At or Above Grade Level 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

At or above grade level (%) 
Deviation from baseline 


-0.7 


-3.9 


11.0 * 


6.1 


2.0 


-0.02 


-0.10 


0.28 * 0.16 


0.05 


School B 

At or above grade level (%) 
Deviation from baseline 


5.6 


5.7 


-3.1 


1.8 


3.7 


0.14 


0.14 


-0.08 0.05 


0.09 


School C 

At or above grade level (%) 
Deviation from baseline 


5.7 


-1.6 


-1.7 


0.1 




0.14 


-0.04 


-0.04 0.00 




School D 

At or above grade level (%) 
Deviation from baseline 


-0.2 


-3.9 


-0.5 


4.6 




-0.01 


-0.10 


-0.01 0.12 




School E 

At or above grade level (%) 
Deviation from baseline 


-0.2 


-1.1 


-3.4 






-0.01 


-0.03 


-0.09 




School F 

At or above grade level (%) 
Deviation from baseline 


3.1 


-2.8 








0.08 


-0.07 








All early-implementing schools 
At or above grade level (%) 
Deviation from baseline 


2.2 


-1.3 


0.4 


3.2 


2.9 


0.06 


-0.03 


0.01 0.08 


0.07 



(continued) 
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TR Table B.8 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes students 
not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4. and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4. and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and the 
18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school estimates, 
which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of each page 
include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table B.9 



Year-by-Year Levels and Impacts for SAT-9 Reading Scores in the Bottom Quartile 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



School Cluster 




Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















In the bottom quartile (%) 


51.7 


46.2 


41.4 


43.7 


45.0 


43.8 


44.8 


50.5 


54.9 


43.5 


Deviation from baseline 


5.7 


0.2 


-4.6 


-2.3 


- 1.0 


-5.3 * 


-4.3 


1.4 


5.8 ** 


-5.6 ** 


School B 






















In the bottom quartile (%) 


38.9 


32.9 


39.4 


36.9 


30.2 


40.2 


38.4 


38.9 


44.4 


36.0 


Deviation from baseline 


-3.5 


-9.4 




-5.5 


- 12.2 


-2.4 


-4.2 


-3.7 


1.8 


- 6.6 


School C 






















In the bottom quartile (%) 


50.3 


69.5 


56.3 


58.9 




52.1 


61.6 


57.4 


54.7 




Deviation from baseline 


-14.2 * 


4.9 


- 8.2 


-5.6 




-6.4 ** 


3.1 


- 1.0 


-3.8 




School D 






















In the bottom quartile (%) 


57.8 


62.7 


63.3 


49.9 




50.2 


50.9 


64.2 


54.6 




Deviation from baseline 


4.7 


9.6 


10.2 


-3.2 




-6.4 


-5.7 


7.6 


- 2.0 




School E 






















In the bottom quartile (%) 


49.0 


52.3 


46.2 






56.3 


59.1 


46.1 






Deviation from baseline 


6.4 


9.7 


3.6 






*** 


10.4 *** 


- 2.6 






School F 






















In the bottom quartile (%) 


55.3 


63.4 








61.2 


58.9 








Deviation from baseline 


-7.1 


1.0 








7 2 *** 


4 9 ** 










All early-implementing schools 






















In the bottom quartile (%) 


50.5 


54.5 


49.3 


47.3 


37.6 


50.6 


52.3 


51.4 


52.2 


39.8 


Deviation from baseline 


-1.3 


2.7 


-0.4 


-4.1 


- 6.6 


-0.9 


0.7 


0.3 


0.5 


- 6.1 ** 
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- 92 - 



The Talent Development Evaluation 
TR Table B.9 (continued) 



Year-by-Year Levels and Impacts for SAT -9 Reading Scores in the Bottom Quartile 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 









Impact 










Impact Effect Size 




School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 Year 4 


Year 5 


School A 

In the bottom quartile (%) 
Deviation from baseline 


11.1 


4.5 


-6.0 


-8.1 


4.6 


0.22 


0.09 


-0.12 -0.16 


0.09 


School B 

In the bottom quartile (%) 
Deviation from baseline 


-1.1 


-5.2 


0.7 


-7.3 


-5.6 


-0.02 


-0.11 


0.01 -0.15 


-0.11 


School C 

In the bottom quartile (%) 
Deviation from baseline 


-7.8 


1.8 


-7.2 


-1.8 




-0.16 


0.04 


-0.14 -0.04 




School D 

In the bottom quartile (%) 
Deviation from baseline 


11.2 


15.3 * 


2.6 


-1.2 




0.22 


0.31 


* 0.05 -0.02 




School E 

In the bottom quartile (%) 
Deviation from baseline 


-1.3 


-0.7 


6.2 






-0.03 


-0.01 


0.12 




School F 

In the bottom quartile (%) 
Deviation from baseline 


-14.3 * 


-4.0 








-0.29 * 


-0.08 








All early-implementing schools 
In the bottom quartile (%) 
Deviation from baseline 


-0.4 


2.0 


-0.7 


-4.6 


-0.5 


-0.01 


0.04 


-0.01 -0.09 


-0.01 



(continued) 
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TR Table B.9 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1. Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and 
the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 

10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school 
estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of 
each page include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 
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The Talent Development Evaluation 
TR Table B.10 

Year-by-Year Levels and Impacts for Attendance Rate 

for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















7th-grade attendance rate (%) 


85.9 


87.2 


88.5 


87.5 


86.9 


87.4 


86.8 


86.3 


86.2 


86.3 


Deviation from baseline 


-1.2 


0.1 


1.3 


0.4 


-0.2 


1.5 * 


0.8 


0.3 


0.3 


0.4 


School B 






















7th-grade attendance rate (%) 


88.0 


88.6 


89.4 


88.1 


89.3 


88.8 


87.9 


87.2 


87.8 


87.5 


Deviation from baseline 


-0.9 


-0.3 


0.5 


-0.7 


0.5 


0.6 


-0.2 


-1.0 


-0.4 


-0.7 


School C 






















7th-grade attendance rate (%) 


82.2 


84.3 


80.1 


76.8 




84.9 


85.0 


84.8 


84.5 




Deviation from baseline 


0.2 


2.3 


-1.9 


-5.2 ** 




0.7 


0.9 


0.7 


0.3 




School D 






















7th-grade attendance rate (%) 


86.1 


86.3 


83.6 


85.0 




82.5 


85.0 


82.6 


86.4 




Deviation from baseline 


0.5 


0.7 


-2.0 


-0.6 




0.7 


3.2 * 


0.8 


4.7 ** 




School E 






















7th-grade attendance rate (%) 


85.2 


88.4 


90.6 






85.7 


85.5 


85.9 






Deviation from baseline 


-2.0 


1.1 


3.4 * 






-0.3 


-0.5 


0.0 







School F 

7th-grade attendance rate (%) 
Deviation from baseline 



All early-implementing schools 

7th-grade attendance rate (%) 85.5 87.0 86.4 84.4 88.1 85.8 86.0 85.3 86.2 86.9 

Deviation from baseline -0.7 0.8 0.3 -1.5 0.1 0.7 0.8 ** 0.2 1.2 ** -0.2 



(continued) 
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The Talent Development Evaluation 
TR Table B.10 (continued) 



Year-by-Year Levels and Impacts for Attendance Rate 

for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

7th-grade attendance rate (%) 
Deviation from baseline 


-2.7 


-0.7 


1.0 


0.1 


-0.6 


-0.09 


-0.02 


0.03 


0.00 


-0.02 


School B 

7th-grade attendance rate (%) 
Deviation from baseline 


-1.5 


0.0 


1.5 


-0.3 


1.2 


-0.05 


0.00 


0.05 


-0.01 


0.04 


School C 

7th-grade attendance rate (%) 
Deviation from baseline 


-0.5 


1.5 


-2.6 


-5.5 ** 




-0.02 


0.05 


-0.09 


-0.18 ** 




School D 

7th-grade attendance rate (%) 
Deviation from baseline 


-0.2 


-2.5 


-2.8 


-5.2 * 




-0.01 


-0.08 


-0.09 


-0.17 * 




School E 

7th-grade attendance rate (%) 
Deviation from baseline 


-1.7 


1.7 


3.5 






-0.06 


0.06 


0.12 







School F 

7th-grade attendance rate (%) 
Deviation from baseline 

























All early-implementing schools 
7th-grade attendance rate (%) 
Deviation from baseline 


-1.3 


0.0 


0.1 


-2.8 ** 


0.3 


-0.04 


0.00 


0.00 


-0.09 ** 


0.01 



(continued) 
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TR Table B.10 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes students 
not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and non 
Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and the 
18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 

Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school estimates, 
which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of each page 
include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a given school 
year. 
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The Talent Development Evaluation 
TR Table B.ll 

Year-by-Year Levels and Impacts for Attendance Rates Greater Than or Equal to 90 Percent 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 



School Cluster 




Talent Development Schools 






Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















Attendance rate of 90% or higher (%) 


50.0 


52.0 


54.6 


49.9 


49.2 


55.4 


51.2 


51.2 


50.8 


50.3 


Deviation from baseline 


0.0 


2.0 


4.6 


-0.1 


-0.8 


3.6 


-0.6 


-0.6 


-1.0 


-1.5 


School B 






















Attendance rate of 90% or higher (%) 


62.1 


60.7 


61.3 


61.7 


63.6 


58.8 


55.1 


52.1 


56.5 


53.8 


Deviation from baseline 


2.6 


1.2 


1.8 


2.2 


4.1 


0.0 


-3.7 


-6.7 ** 


-2.3 


-5.0 


School C 






















Attendance rate of 90% or higher (%) 


34.1 


42.0 


41.8 


35.0 




46.8 


46.4 


44.2 


43.6 




Deviation from baseline 


-1.8 


6.1 


6.0 


-0.9 




1.7 


1.3 


-0.9 


-1.5 




School D 






















Attendance rate of 90% or higher (%) 


44.4 


49.3 


40.5 


45.1 




36.1 


43.9 


35.1 


51.1 




Deviation from baseline 


-2.0 


2.8 


-6.0 


-1.3 




-4.0 


3.8 


-5.0 


11.0 ** 




School E 






















Attendance rate of 90% or higher (%) 


43.9 


53.5 


65.4 






49.7 


47.3 


49.3 






Deviation from baseline 


-13.6 ** 


-4.0 


7.9 






-0.9 


-3.2 


-1.3 







School F 

Attendance rate of 90% or higher (%) 
Deviation from baseline 



All early-implementing schools 

Attendance rate of 90% or higher (%) 46.9 51.5 52.7 47.9 56.4 49.4 48.8 46.4 50.5 52.1 

Deviation from baseline -2.9 1.6 2.9 0.0 1.7 0.1 -0.5 -2.9 ** 1.6 -3.2 * 



(continued) 
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The Talent Development Evaluation 
TR Table B.ll (continued) 

Year-by-Year Levels and Impacts for Attendance Rates Greater Than or Equal to 90 Percent 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


-3.6 


2.7 


5.2 


0.8 


0.7 


-0.07 


0.05 


0.11 


0.02 


0.01 


School B 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


2.7 


4.9 


8.5 


4.5 


9.1 * 


0.05 


0.10 


0.17 


0.09 


0.18 * 


School C 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


-3.5 


4.8 


6.8 


0.6 




-0.07 


0.10 


0.14 


0.01 




School D 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


2.0 


-0.9 


-1.0 


-12.3 




0.04 


-0.02 


-0.02 


-0.25 




School E 

Attendance rate of 90% or higher (%) 
Deviation from baseline 


-12.8 * 


-0.8 


9.1 






-0.26 * 


-0.02 


0.18 







School F 

Attendance rate of 90% or higher (%) 
Deviation from baseline 



All early-implementing schools 
Attendance rate of 90% or higher (%) 

Deviation from baseline -3.0 2.1 5.7 * -1.6 4.9 -0.06 0.04 0.12 * -0.03 0.10 



(continued) 
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TR Table B.ll (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes students 
not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4. and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and the 
18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school estimates, 
which represent the average of several schools, as compared to individual Talent Development school estimates. Similarly, across-school averages at the bottom of each page 
include a larger sample of schools in the first three follow-up years as compared Year 4 and Year 5. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a given 
school year. 
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The Talent Development Evaluation 
TR Table B.12 

Year-by-Year Levels and Impacts for Attendance Rates Less Than or Equal to 80 Percent 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 






















Attendance rate of 80% or lower (%) 


18.1 


22.6 


21.8 


23.5 


24.2 


19.5 


24.8 


24.1 


25.4 


25.0 


Deviation from baseline 


-5.1 


-0.6 


-1.4 


0.3 


1.0 


-4.7 ** 


0.6 


-0.2 


1.1 


0.7 


School B 






















Attendance rate of 80% or lower (%) 


17.6 


14.7 


17.3 


18.3 


14.9 


16.6 


21.9 


22.3 


22.6 


23.7 


Deviation from baseline 


2.0 


-0.9 


1.7 


2.7 


-0.7 


-2.8 


2.4 


2.8 


3.2 


4.3 * 


School C 






















Attendance rate of 80% or lower (%) 


41.5 


28.5 


35.4 


36.9 




28.9 


27.3 


27.6 


28.8 




Deviation from baseline 


4.7 


-8.3 


-1.4 


0.1 




0.2 


-1.4 


-1.1 


0.2 




School D 






















Attendance rate of 80% or lower (%) 


22.6 


21.9 


34.4 


28.2 




35.2 


30.5 


36.0 


27.1 




Deviation from baseline 


0.0 


-0.8 


11.7 * 


5.5 




1.7 


-3.0 


2.5 


-6.4 




School E 






















Attendance rate of 80% or lower (%) 


23.6 


19.6 


13.4 






26.0 


27.2 


25.2 






Deviation from baseline 


1.8 


-2.2 


-8.3 






1.3 


2.5 


0.5 







School F 

Attendance rate of 80% or lower (%) 
Deviation from baseline 



All early-implementing schools 

Attendance rate of 80% or lower (%) 24.7 21.5 24.5 26.7 19.6 25.2 26.3 27.0 26.0 24.4 

Deviation from baseline 0.7 -2.5 0.5 2.2 0.2 -0.9 0.2 0.9 -0.5 2.5 * 



(continued) 
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The Talent Development Evaluation 
TR Table B.12 (continued) 



Year-by-Year Levels and Impacts for Attendance Rates Less Than or Equal to 80 Percent 
for Seventh-Grade Students in Early-Implementing Talent Development Schools and Their Comparison Schools, 

Five-Year Follow-Up Results, by School Cluster 



Impact Impact Effect Size 



School Cluster 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


School A 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


-0.4 


-1.2 


-1.2 


-0.8 


0.3 


-0.01 


-0.03 


-0.03 


-0.02 


0.01 


School B 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


4.8 


-3.3 


-1.2 


-0.4 


-5.0 


0.10 


-0.07 


-0.03 


-0.01 


-0.11 


School C 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


4.5 


-6.8 


-0.3 


0.0 




0.10 


-0.15 


-0.01 


0.00 




School D 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


-1.8 


2.2 


9.3 


11.9 




-0.04 


0.05 


0.20 


0.26 




School E 

Attendance rate of 80% or lower (%) 
Deviation from baseline 


0.5 


-4.7 


-8.9 






0.01 


-0.10 


-0.19 







School F 

Attendance rate of 80% or lower (%) 
Deviation from baseline 



All early-implementing schools 
Attendance rate of 80% or lower (%) 

Deviation from baseline 1.5 -2.8 -0.5 2.7 -2.3 0.03 -0.06 -0.01 0.06 -0.05 



(continued) 
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TR Table B.12 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 6 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes students 
not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of between 2 and 1 1 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

The deviation from the baseline for Year 1, Year 2, Year 3, Year 4, and Year 5 was calculated as the difference between the baseline average and the Year 1, Year 2, Year 3, 
Year 4, and Year 5 averages, respectively. 

The impacts for Year 1, Year 2, Year 3, Year 4, and Year 5 were calculated as the difference in deviations from baseline average between Talent Development schools and 
non-Talent Development schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools and the 
18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Blank spaces under the Year 4 and Year 5 columns indicate that, at the time of analysis, some clusters had not yet completed a fourth or fifth year of implementation or data 
were not available for that outcome. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated a prior 
grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 

Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent Development school estimates, 
which represent the average of several schools, as compared with individual Talent Development school estimates. Similarly, across-school averages at the bottom of each page 
include a larger sample of schools in the first three follow-up years, as compared with Year 4 and Year 5. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was enrolled in a given school 
year. 
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The Talent Development Evaluation 
TR Table C.l 



Year-by-Year Levels and Impacts for Math NCE Scores 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One- Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

8 th-grade average NCE 


28.5 


31.8 


3.2 


31.3 


33.0 


1.7 


1.5 


0.10 


School H 

8 th-grade average NCE 


29.7 


29.4 


-0.4 


26.1 


28.6 


2.5 *** 


-2.9 


- 0.20 


School 1 

8 th-grade average NCE 


26.2 


33.1 


6.9 ** 


24.6 


27.6 


2 9 ** 


3.9 


0.27 


School J 

8 th-grade average NCE 


26.9 


31.3 


4.4 


25.3 


26.8 


1.5 


2.9 


0.20 


School K 

8 th-grade average NCE 


27.0 


29.2 


2.1 


28.4 


29.6 


1.3 * 


0.9 


0.06 





All later-implementing schools 
8 th-grade average NCE 


27.7 


30.9 


3 3 *** 


27.1 


29.1 


2 0 *** 


1.3 


0.09 



(continued) 
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TR Table C.l (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 
Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared wtih individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.2 



Year-by-Year Levels and Impacts for Math Scores At or Above Grade Level 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

At or above grade level (%) 


9.6 


9.2 


-0.4 


11.2 


11.9 


0.7 


-1.1 


-0.05 


School H 

At or above grade level (%) 


8.7 


10.0 


1.3 


6.0 


6.0 


-0.1 


1.4 


0.06 


School I 

At or above grade level (%) 


5.1 


9.4 


4.3 


4.2 


3.7 


-0.5 


4.8 


0.22 


School J 

At or above grade level (%) 


7.3 


10.7 


3.3 


6.1 


3.7 


-2.4 


5.7 


0.26 


School K 

At or above grade level (%) 


6.8 


5.4 


-1.4 


8.2 


7.7 


-0.5 


-0.9 


-0.04 




All later-implementing schools 
At or above grade level (%) 


7.5 


8.9 


1.4 


7.1 


6.6 


-0.5 


2.0 


0.09 



(continued) 
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TR Table C.2 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 
Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the "Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.3 



Year-by-Year Levels and Impacts for Math Scores in the Bottom Quartile 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One- Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

In the bottom quartile (%) 


72.5 


67.4 


-5.1 


64.3 


58.9 


-5.4 


0.3 


0.01 


School H 

In the bottom quartile (%) 


68.8 


65.7 


-3.0 


77.5 


72.9 


-4.6 * 


1.6 


0.04 


School I 

In the bottom quartile (%) 


76.1 


56.8 


-19.3 *** 


80.9 


76.3 


-4.6 ** 


-14.7 ** 


-0.37 ** 


School J 

In the bottom quartile (%) 


73.0 


72.8 


-0.2 


77.4 


80.1 


2.7 


-2.9 


-0.07 


School K 

In the bottom quartile (%) 


76.1 


72.3 


-3.8 


71.6 


68.9 


-2.6 


-1.2 


-0.03 





All later-implementing schools 
In the bottom quartile (%) 


73.3 


67.0 


-6.3 ** 


74.3 


71.4 


2 9 ** 


-3.4 


-0.09 



(continued) 
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TR Table C.3 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1" columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.4 



Year-by-Year Levels and Impacts for Reading NCE Scores 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

8 th-grade average NCE 


29.4 


30.4 


1.0 


31.7 


34.0 


2.4 ** 


-1.4 


-0.09 


School H 

8 th-grade average NCE 


25.1 


27.3 


2.3 


27.6 


30.9 


3 3 *** 


- 1.0 


-0.06 


School I 

8 th-grade average NCE 


26.0 


30.7 


4.7 


26.0 


30.6 


4 7 *** 


0.0 


0.00 


School J 

8 th-grade average NCE 


28.6 


35.3 


6.6 ** 


26.6 


25.5 


- 1.1 


7 7 ** 


0.48 ** 


School K 

8 th-grade average NCE 


30.4 


30.6 


0.2 


29.8 


31.7 


1.9 ** 


-1.7 


- 0.11 





All later-implementing schools 
8 th-grade average NCE 


27.9 


30.9 


2 9 *** 


28.3 


30.5 


2 2 *** 


0.7 


0.05 



(continued) 
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TR Table C.4 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the "Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.5 



Year-by-Year Levels and Impacts for Reading Scores At or Above Grade Level 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

At or above grade level (%) 


9.0 


5.8 


-3.3 


12.9 


11.6 


-1.4 


-1.9 


-0.07 


School H 

At or above grade level (%) 


5.0 


4.8 


-0.2 


7.8 


7.9 


0.1 


-0.3 


-0.01 


School I 

At or above grade level (%) 


6.0 


9.3 


3.3 


6.3 


6.5 


0.2 


3.1 


0.11 


School J 

At or above grade level (%) 


11.2 


13.5 


2.3 


6.1 


3.9 


-2.2 


4.5 


0.16 


School K 

At or above grade level (%) 


10.7 


9.3 


-1.4 


10.0 


9.2 


-0.8 


-0.6 


-0.02 




All later-implementing schools 
At or above grade level (%) 


8.4 


8.5 


0.1 


8.6 


7.8 


-0.8 


1.0 


0.03 



(continued) 



-PI l- 



TR Table C.5 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.6 



Year-by-Year Levels and Impacts for Reading Scores in the Bottom Quartile 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 



School Cluster 



Talent Development Schools 
Baseline Year 1 Difference 



Non-Talent Development Schools Impact 

Baseline Year 1 Difference Impact Effect Size 



School G 



In 


69.3 


67.6 


-1.7 


65.2 


58.7 


-6.4 * 


4.7 


0.10 


School H 

In the bottom quartile (%) 


78.0 


78.6 


0.6 


74.2 


68.5 


-5.7 ** 


6.3 


0.14 


School I 

In the bottom quartile (%) 


76.3 


69.4 


-7.0 


77.6 


68.5 


_9 i *** 


2.2 


0.05 


School I 

In the bottom quartile (%) 


72.5 


54.0 


-18.4 ** 


73.8 


80.2 


6.4 


-24.8 ** 


-0.54 


School K 

In the bottom quartile (%) 


68.2 


70.1 


1.8 


69.6 


65.6 


-4.0 


5.8 


0.13 





All later-implementing schools 
In the bottom quartile (%) 


72.9 


67.9 


-4.9 * 


72.1 


68.3 


-3.8 *** 


-1.2 


-0.03 



(continued) 
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TR Table C.6 (continued) 

SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1 . 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 

** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.7 



Year-by-Year Levels and Impacts for Attendance Rate 

for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

8th-grade attendance rate (%) 


88.1 


88.5 


0.4 


86.5 


87.7 


1.3 


-0.9 


-0.03 


School H 

8th-grade attendance rate (%) 


84.7 


86.0 


1.3 


85.7 


85.2 


-0.5 


1.8 


0.06 


School 1 

8th-grade attendance rate (%) 


81.7 


82.4 


0.7 


85.4 


85.1 


-0.3 


1.0 


0.03 


School J 

8th-grade attendance rate (%) 


82.3 


84.2 


1.9 


83.9 


81.6 


-2.3 


4.3 * 


0.14 * 


School K 

8th-grade attendance rate (%) 


86.2 


85.4 


-0.7 


86.5 


86.6 


0.2 


-0.9 


-0.03 




All later-implementing schools 
8th-grade attendance rate (%) 


84.6 


85.3 


0.7 


85.6 


85.3 


-0.3 


1.1 


0.04 



(continued) 
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TR Table C.7 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.8 



Year-by-Year Levels and Impacts for Attendance of 90 Percent or Higher 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

Attendance rate of 90% or higher (%) 


59.7 


64.1 


4.4 


53.9 


58.4 


4.5 


-0.1 


0.00 


School H 

Attendance rate of 90% or higher (%) 


47.3 


48.1 


0.8 


50.8 


49.7 


-1.0 


1.8 


0.04 


School I 

Attendance rate of 90% or higher (%) 


35.1 


43.5 


8.4 


50.4 


49.8 


-0.6 


9.0 


0.18 


School J 

Attendance rate of 90% or higher (%) 


41.3 


46.9 


5.6 


40.9 


38.9 


-2.0 


7.6 


0.16 


School K 

Attendance rate of 90% or higher (%) 


52.3 


50.7 


-1.5 


52.7 


54.1 


1.4 


-2.9 


-0.06 




All later-implementing schools 

Attendance rate of 90% or higher (%) 


47.1 


50.7 


3.5 


49.8 


50.2 


0.4 


3.1 


0.06 



(continued) 
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TR Table C.8 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the "Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the "Difference" columns reflect the difference in deviations from the baseline average and the average in Year 1 . 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 

** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.9 



Year-by-Year Levels and Impacts for Attendance Rates Less Than or Equal to 80 Percent 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

Attendance rate of 80% or lower (%) 


20.3 


18.9 


-1.4 


23.2 


21.9 


-1.3 


-0.2 


0.00 


School H 

Attendance rate of 80% or lower (%) 


29.1 


24.5 


-4.6 


24.4 


26.0 


1.5 


-6.2 


-0.13 


School 1 

Attendance rate of 80% or lower (%) 


37.9 


30.8 


-7.2 


25.0 


26.5 


1.5 


-8.7 


-0.18 


School J 

Attendance rate of 80% or lower (%) 


33.9 


28.0 


-5.9 


29.5 


31.3 


1.8 


-7.7 


-0.16 


School K 

Attendance rate of 80% or lower (%) 


24.0 


24.0 


-0.1 


23.0 


22.8 


-0.2 


0.2 


0.00 




All later-implementing schools 

Attendance rate of 80% or lower (%) 


29.1 


25.2 


oo 

* 


25.0 


25.7 


0.7 


-4.5 ** 


-0.09 ** 



(continued) 
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TR Table C.9 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 
Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table C.10 



Year-by-Year Levels and Impacts for One- Year Promotion Rates 
for Eighth-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One- Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

Promoted to 9th grade 


95.9 


94.8 


-1.1 


97.3 


98.6 


1.4 


-2.4 


-0.10 


School H 

Promoted to 9th grade 


99.1 


97.8 


-1.2 


97.8 


97.6 


-0.2 


-1.1 


-0.05 


School I 

Promoted to 9th grade 


98.8 


99.3 


0.6 


98.1 


97.0 


-1.2 


1.7 


0.07 


School J 

Promoted to 9th grade 


92.3 


101.8 


9.5 


96.2 


89.8 


-6.5 


15.9 * 


0.68 * 


School K 

Promoted to 9th grade 


98.7 


94.6 


-4.0 * 


97.5 


98.4 


0.9 


-5.0 ** 


-0.21 ** 





All later-implementing schools 
Promoted to 9th grade 


96.9 


97.7 


0.7 


97.4 


96.3 


-1.1 


1.8 


0.08 



(continued) 
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TR Table C.10 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 8th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 8th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 

** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.l 



Year-by-Year Levels and Impacts for SAT-9 Math Total NCE Scores 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

7th-grade average NCE 


37.8 


39.8 


2.0 


36.8 


41.1 


4.3 ** 


-2.3 


-0.16 


School H 

7th-grade average NCE 


33.9 


42.0 


00 

* 

* 


32.8 


35.2 


2 4 ** 


5.6 


0.39 


School I 

7th-grade average NCE 


31.5 


37.0 


5.6 * 


31.0 


33.3 


2.4 * 


3.2 


0.22 


School J 

7th-grade average NCE 


36.4 


39.3 


2.9 


33.6 


37.0 


3.4 


-0.5 


-0.04 


School K 

7th-grade average NCE 


34.6 


35.2 


0.6 


34.9 


37.1 


2.3 * 


-1.6 


-0.11 





All later-implementing schools 
7th-grade average NCE 


34.8 


38.7 


3.8 ** 


33.8 


36.8 


3 0 *** 


0.9 


0.06 



(continued) 
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TR Table D.l (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Numbers in the ‘'Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.2 



Year-by-Year Levels and Impacts for SAT-9 Math Total Scores At or Above Grade Level 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

At or above grade level (%) 


19.1 


21.2 


2.1 


19.2 


27.0 


7.8 * 


-5.7 


-0.18 


School H 

At or above grade level (%) 


12.0 


28.9 


16.9 * 


11.0 


13.6 


2.6 


14.3 


0.44 


School I 

At or above grade level (%) 


8.7 


15.0 


6.3 


8.7 


8.5 


-0.1 


6.4 


0.20 


School J 

At or above grade level (%) 


16.5 


18.9 


2.4 


11.6 


13.9 


2.3 


0.2 


0.00 


School K 

At or above grade level (%) 


14.7 


13.9 


-0.8 


14.8 


18.4 


3.5 


-4.3 


-0.13 




All later-implementing schools 
At or above grade level (%) 


14.2 


19.6 


5.4 


13.1 


16.3 


3.2 * 


2.2 


0.07 



(continued) 
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TR Table D.2 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.3 



Year-by-Year Levels and Impacts for SAT-9 Math Total Scores in the Bottom Quartile 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

In the bottom quartile (%) 


50.1 


44.7 


-5.3 


53.0 


41.2 


-11.8 ** 


6.5 


0.14 


School H 

In the bottom quartile (%) 


63.3 


40.9 


-22.3 * 


64.5 


60.1 


-4.5 


-17.9 


-0.38 


School I 

In the bottom quartile (%) 


67.8 


54.2 


-13.6 


69.2 


66.7 


-2.5 


-11.1 


-0.23 


School J 

In the bottom quartile (%) 


53.3 


42.0 


-11.3 


62.4 


52.1 


-10.3 


-1.0 


-0.02 


School K 

In the bottom quartile (%) 


57.9 


56.8 


-1.2 


58.5 


53.0 


-5.6 


4.4 


0.09 





All later-implementing schools 
In the bottom quartile (%) 


58.5 


47.7 


-10.8 ** 


61.5 


54.6 


_6 9 *** 


-3.8 


-0.08 



(continued) 
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TR Table D.3 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the "Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the "Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 
percent; ** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for 
non-Talent Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.4 



Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving NCE Scores 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

7th-grade average NCE 


38.5 


40.7 


2.2 


37.3 


41.5 


4.2 ** 


-2.0 


-0.13 


School H 

7th-grade average NCE 


33.6 


41.7 


8.1 ** 


33.2 


35.8 


2.6 ** 


5.4 


0.36 


School I 

7th-grade average NCE 


31.7 


38.7 


7.0 ** 


31.5 


33.6 


2.1 


4.9 


0.32 


School J 

7th-grade average NCE 


36.6 


39.8 


3.2 


35.4 


32.1 


-3.4 


6.6 * 


0.43 * 


School K 

7th-grade average NCE 


33.7 


36.4 


2.7 


35.2 


37.6 


2.4 ** 


0.3 


0.02 





All later-implementing schools 
7th-grade average NCE 


34.8 


39.5 


4 6 *** 


34.5 


36.1 


1.6 ** 


3.0 * 


0.20 * 



(continued) 






- 134 - 



TR Table D.4 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the ‘'Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.5 



Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving Scores At or Above Grade Level 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

At or above grade level (%) 


22.1 


22.1 


0.0 


20.1 


29.2 


9.2 ** 


-9.2 


-0.29 


School H 

At or above grade level (%) 


11.6 


26.9 


15.3 * 


11.6 


16.0 


4.4 * 


11.0 


0.34 


School I 

At or above grade level (%) 


10.3 


18.6 


8.3 


9.4 


10.7 


1.3 


7.0 


0.22 


School J 

At or above grade level (%) 


17.5 


21.1 


3.6 


15.4 


8.7 


-6.7 


10.3 


0.32 


School K 

At or above grade level (%) 


14.0 


17.8 


3.8 


15.5 


20.3 


4.9 * 


-1.1 


-0.03 




All later-implementing schools 
At or above grade level (%) 


15.1 


21.3 


6.2 * 


14.4 


17.0 


2.6 


3.6 


0.11 



(continued) 
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TR Table D.5 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the "Year 1" columns reflect averages for the first year of Talent Development implementation. 

Numbers in the "Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.6 



Year-by-Year Levels and Impacts for SAT-9 Math Problem Solving Scores in the Bottom Quartile 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

In the bottom quartile (%) 


48.0 


37.9 


-10.1 


51.4 


41.4 


-10.0 ** 


-0.1 


0.00 


School H 

In the bottom quartile (%) 


65.5 


38.6 


-26.9 ** 


62.7 


55.8 


-6.9 ** 


-20.0 * 


-0.42 * 


School I 

In the bottom quartile (%) 


65.9 


45.3 


-20.6 ** 


67.0 


62.1 


-4.9 


-15.7 


-0.33 


School J 

In the bottom quartile (%) 


53.4 


41.6 


-11.8 


55.8 


67.5 


11.7 * 


-23.5 ** 


-0.50 ** 


School K 

In the bottom quartile (%) 


57.9 


53.2 


-4.8 


57.2 


51.2 


-6.0 ** 


1.2 


0.03 





All later-implementing schools 
In the bottom quartile (%) 


58.1 


43.3 


-14.8 *** 


58.8 


55.6 


-3.2 ** 


-11.6 *** 


3.25 *** 



(continued) 
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TR Table D.6 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1 . 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.7 



Year-by-Year Levels and Impacts for SAT-9 Reading NCE Scores 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

7th-grade average NCE 


39.3 


42.3 


3.0 


39.0 


42.7 


3.8 ** 


- 0.8 


-0.05 


School H 

7th-grade average NCE 


31.1 


41.4 


10.2 *** 


34.9 


37.7 


2 g *** 


7.4 ** 


0.43 ** 


School I 

7th-grade average NCE 


33.7 


41.1 


73 ** 


34.4 


35.9 


1.5 


5.8 * 


0.34 * 


School J 

7th-grade average NCE 


38.0 


38.6 


0.6 


35.5 


32.3 


-3.2 


3.8 


0.22 


School K 

7th-grade average NCE 


36.5 


38.5 


2.0 


36.5 


39.9 


3.4 *** 


-1.4 


-0.08 





All later-implementing schools 
7th-grade average NCE 


35.7 


40.4 


4 6 *** 


36.0 


37.7 


\ 7 ** 


3.0 * 


0.17 * 



(continued) 
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TR Table D.7 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1 . 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.8 



Year-by-Year Levels and Impacts for SAT-9 Reading Scores At or Above Grade Level 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

At or above grade level (%) 


26.8 


25.0 


-1.8 


26.9 


29.7 


2.8 


-4.5 


-0.11 


School H 

At or above grade level (%) 


14.2 


30.5 


16.3 ** 


18.8 


20.2 


1.5 


14.8 * 


0.37 * 


School I 

At or above grade level (%) 


14.9 


26.1 


1 1.2 ** 


18.0 


16.2 


-1.9 


13.1 ** 


0.33 ** 


School J 

At or above grade level (%) 


25.4 


27.3 


1.9 


20.2 


8.3 


-11.9 * 


13.8 


0.35 


School K 

At or above grade level (%) 


21.3 


14.8 


-6.5 


21.8 


24.2 


2.4 


-8.9 


-0.23 




All later-implementing schools 
At or above grade level (%) 


20.5 


24.7 


4.2 


21.1 


19.7 


-1.4 


5.6 


0.14 
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- 142 - 



TR Table D.8 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 

** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.9 



Year-by-Year Levels and Impacts for SAT-9 Reading Scores in the Bottom Quartile 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

In the bottom quartile (%) 


42.4 


37.1 


-5.3 


43.8 


37.2 


-6.6 


1.3 


0.03 


School H 

In the bottom quartile (%) 


62.5 


46.1 


-16.4 * 


55.0 


49.3 


-5.7 ** 


-10.7 


-0.22 


School I 

In the bottom quartile (%) 


57.9 


37.6 


-20.3 ** 


55.9 


54.5 


-1.4 


-18.9 ** 


-0.38 ** 


School J 

In the bottom quartile (%) 


44.7 


44.0 


-0.7 


51.3 


62.0 


10.7 


-11.3 


-0.23 


School K 

In the bottom quartile (%) 


51.8 


41.4 


-10.4 


50.5 


43.7 


-6.7 ** 


-3.7 


-0.07 





All later-implementing schools 
In the bottom quartile (%) 


51.9 


41.2 


-10.6 *** 


51.3 


49.4 


-2.0 


-8.7 * 


-0.17 * 



(continued) 
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TR Table D.9 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample 
includes students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given 
school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 

Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact 
estimates. Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.10 

Year-by-Year Levels and Impacts for Attendance Rate 

for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One- Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

7th-grade attendance rate (%) 


88.6 


87.6 


-1.0 


86.9 


87.0 


0.1 


-1.1 


-0.04 


School H 

7th-grade attendance rate (%) 


85.0 


86.9 


1.9 


85.4 


85.1 


-0.3 


2.2 


0.07 


School I 

7th-grade attendance rate (%) 


83.2 


83.2 


0.0 


85.0 


84.5 


-0.5 


0.4 


0.01 


School J 

7th-grade attendance rate (%) 


83.0 


81.8 


-1.2 


84.3 


85.4 


1.2 


-2.4 


-0.08 


School K 

7th-grade attendance rate (%) 


85.6 


85.7 


0.1 


86.5 


86.4 


-0.1 


0.2 


0.01 





All later-implementing schools 
7th-grade attendance rate (%) 


85.1 


85.0 


-0.1 


85.6 


85.7 


0.1 


-0.1 


0.00 



(continued) 
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TR Table D.10 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development schools 
were counted in more than one cluster. 

Numbers in the ‘'Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.ll 

Year-by-Year Levels and Impacts for Attendance Rates Greater Than or Equal to 90 Percent 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

Attendance rate of 90% or higher (%) 


58.4 


58.7 


0.3 


53.7 


53.4 


-0.2 


0.5 


0.01 


School H 

Attendance rate of 90% or higher (%) 


44.2 


56.3 


12.1 * 


47.7 


46.1 


-1.6 


13.7 * 


0.28 * 


School I 

Attendance rate of 90% or higher (%) 


41.0 


43.3 


2.3 


46.3 


44.0 


-2.3 


4.6 


0.09 


School J 

Attendance rate of 90% or higher (%) 


38.3 


28.9 


-9.5 


38.8 


49.8 


11.0 * 


-20.4 ** 


-0.41 ** 


School K 

Attendance rate of 90% or higher (%) 


45.8 


51.4 


5.6 


51.5 


50.5 


-0.9 


6.5 


0.13 




All later-implementing schools 

Attendance rate of 90% or higher (%) 


45.6 


47.7 


2.2 


47.6 


48.8 


1.2 


1.0 


0.02 



(continued) 
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TR Table D.ll (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the “Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the "Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1. 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development schools 
and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997.7. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had repeated 
a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 
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The Talent Development Evaluation 
TR Table D.12 

Year-by-Year Levels and Impacts for Attendance Rates Less Than or Equal to 80 Percent 
for Seventh-Grade Students in Later-Implementing Talent Development Schools and Their Comparison Schools, 

One-Year Follow-Up Results, by School Cluster 





Talent Development Schools 


Non-Talent Development Schools 




Impact 


School Cluster 


Baseline 


Year 1 


Difference 


Baseline 


Year 1 


Difference 


Impact 


Effect Size 


School G 

Attendance rate of 80% or lower (%) 


16.3 


20.0 


3.7 


22.8 


23.4 


0.5 


3.1 


0.07 


School H 

Attendance rate of 80% or lower (%) 


28.5 


24.2 


-4.3 


26.7 


26.9 


0.2 


-4.5 


-0.10 


School I 

Attendance rate of 80% or lower (%) 


34.1 


34.4 


0.3 


27.6 


28.5 


0.9 


-0.6 


-0.01 


School J 

Attendance rate of 80% or lower (%) 


34.7 


34.6 


-0.1 


32.0 


30.0 


-1.9 


1.9 


0.04 


School K 

Attendance rate of 80% or lower (%) 


25.8 


24.4 


-1.5 


24.0 


24.4 


0.4 


-1.8 


-0.04 





All later-implementing schools 

Attendance rate of 80% or lower (%) 


27.9 


27.5 


-0.4 


26.6 


26.6 


0.0 


-0.4 


-0.01 
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- 150 - 



TR Table D.12 (continued) 



SOURCE: MDRC calculations from individual students’ school records from a large, urban school district. 

NOTES: Sample includes 7th-grade students from 5 Talent Development middle schools and 18 non-Talent Development middle schools. The analysis sample includes 
students not designated as ESOL or special education for whom a test score record is available or who were enrolled for at least 145 days during a given school year. 

Each school cluster consists of a Talent Development school matched with a group of 1 to 12 non-Talent Development schools. Some non-Talent Development 
schools were counted in more than one cluster. 

Numbers in the "Baseline” columns reflect averages over a three-year period prior to the initial implementation of Talent Development for a given school cluster. 
Numbers in the “Year 1” columns reflect averages for the first year of Talent Development implementation. 

Numbers in the “Difference” columns reflect the difference in deviations from the baseline average and the average in Year 1 . 

The impacts for Year 1 were calculated as the difference in deviations from baseline average between Talent Development schools and non-Talent Development 
schools. 

The impact effect size was calculated by dividing the impact by the standard deviation of the outcome for all 7th-grade students in the 1 1 Talent Development 
schools and the 18 non-Talent Development comparison schools from school years 1995-1996 through 1996-1997. 

Estimates are regression-adjusted using ordinary least squares, controlling for 4th-grade math and reading SAT-9 test scores, race, and whether a student had 
repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development and non-Talent Development comparison schools, and to the impact estimates. 
Standard errors and statistical significance levels are adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. Statistical significance, which in part depends on sample size, may be achieved with deviations of a smaller magnitude for non-Talent 
Development school estimates, which represent the average of several schools, as compared with individual Talent Development school estimates. 




